3:56 p.m.
Dear All,
I'm trying to use the functions sd(x), and mean(x) in a list instead
of a series. From what I understood, these functions ignore NAs when
used in a series, but when used in a list they return NA if any
missing value.
I need to compute the cross-sectional mean and standard deviation and I
want NAs to be ignored, as they are in a series. Is there a simple way
of doing this? Basically a I have a list with 10 series and I want to
compute the standard deviations of all first observations, the standard
deviation of all second observations and so on. In R there is a way of
skipping NAs. Is there anything in Gretl?
Thanks in advance
Filipe
4:20 p.m.
Hi,
you could try to select a joint sample with no-missings
<hansl>
list L = Y X
smpl L --no-missing
summary L
</hansl>
However, this will select a joint set of valid observations. Otherwise
you loop over L
<hansl>
list L = Y X
loop foreach i L -q
smpl $i --no-missing --replace
eval sd($i)
eval mean($i)
endloop
</hansl>
Best,
Artur
Am 03.06.2017 um 17:56 schrieb Filipe Rodrigues da Costa:
Dear All,
I'm trying to use the functions sd(x), and mean(x) in a list instead of
a series. From what I understood, these functions ignore NAs when used
in a series, but when used in a list they return NA if any missing value.
I need to compute the cross-sectional mean and standard deviation and I
want NAs to be ignored, as they are in a series. Is there a simple way
of doing this? Basically a I have a list with 10 series and I want to
compute the standard deviations of all first observations, the standard
deviation of all second observations and so on. In R there is a way of
skipping NAs. Is there anything in Gretl?
Thanks in advance
Filipe
4:53 p.m.
Am 03.06.2017 um 17:56 schrieb Filipe Rodrigues da Costa:
I'm trying to use the functions sd(x), and mean(x) in a list
instead of
a series. From what I understood, these functions ignore NAs when used
in a series, but when used in a list they return NA if any missing value.
Hi,
have you actually verified that this is really what's happening? In the
guide I see the following explanation:
"The functions max, min, mean, sd, sum and var behave “horizontally”
rather than “vertically” when their argument is a list. For instance,
the following commands
list Xlist = x1 x2 x3
series m = mean(Xlist)
produce a series m whose i-th element is the average of x 1,i ,x 2,i and
x 3,i ; missing values, if any, are implicitly discarded."
To me that sounds as if it works as you intend.
cheers,
sven
9:28 a.m.
Hi Sven,
I verified this and in fact, missing values aren't ignored. If you have
just a single missing value in one of the series that is part of the
list, the resulting mean or sd is NA. So, for example, if you have three
series: A, B, and C, with values 100, 200 and NA for 1960, the resulting
mean for a list with A, B and C is NA, and not 150, as desirable.
Here is the transcript from the function reference:
"If x is a list, returns a series y such that yt is the sample standard
deviation of the values of the variables in the list at observation t,
or NA if there are any missing values at t. "
It seems that the guide and function reference are not in accordance.
--
*Filipe Rodrigues da Costa*
Send me an email to: filipe(a)pobox.io
Reach me through *Telegram* at: https://t.me/rodriguesdacosta
On Sat, 3 Jun 2017, at 17:53, Sven Schreiber wrote:
Am 03.06.2017 um 17:56 schrieb Filipe Rodrigues da Costa:
> I'm trying to use the functions sd(x), and mean(x) in a list instead of
> a series. From what I understood, these functions ignore NAs when used
> in a series, but when used in a list they return NA if any missing value.
Hi,
have you actually verified that this is really what's happening? In the
guide I see the following explanation:
"The functions max, min, mean, sd, sum and var behave “horizontally”
rather than “vertically” when their argument is a list. For instance,
the following commands
list Xlist = x1 x2 x3
series m = mean(Xlist)
produce a series m whose i-th element is the average of x 1,i ,x 2,i and
x 3,i ; missing values, if any, are implicitly discarded."
To me that sounds as if it works as you intend.
cheers,
sven
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
9:39 a.m.
Let's get a concrete example:
<hansl>
set verbose off
open denmark.gdt -q
LRM[1] = NA
mean_LRM = mean(LRM)
sd_LRM = sd(LRM)
print mean_LRM sd_LRM
summary LRM --simple
list L = LRM LRY
summary L --simple
</hansl>
yields
<output>
mean_LRM = 11.756663
sd_LRM = 0.15285752
Summary statistics, using the observations 1974:1 - 1987:3
for the variable 'LRM' (54 valid observations)
Mean 11.757
Minimum 11.582
Maximum 12.075
Standard deviation 0.15286
Missing obs. 1
Mean Median S.D. Min Max
LRM 11.76 11.71 0.1529 11.58 12.08
LRY 5.952 5.940 0.07276 5.787 6.099
</output>
Artur
Am 05.06.2017 um 11:28 schrieb Filipe Rodrigues da Costa:
Hi Sven,
I verified this and in fact, missing values aren't ignored. If you have
just a single missing value in one of the series that is part of the
list, the resulting mean or sd is NA. So, for example, if you have three
series: A, B, and C, with values 100, 200 and NA for 1960, the resulting
mean for a list with A, B and C is NA, and not 150, as desirable.
Here is the transcript from the function reference:
"If x is a list, returns a series y such that yt is the sample standard
deviation of the values of the variables in the list at observation t,
or NA if there are any missing values at t. "
It seems that the guide and function reference are not in accordance.
--
*Filipe Rodrigues da Costa*
Send me an email to: filipe(a)pobox.io
Reach me through *Telegram* at: https://t.me/rodriguesdacosta
On Sat, 3 Jun 2017, at 17:53, Sven Schreiber wrote:
> Am 03.06.2017 um 17:56 schrieb Filipe Rodrigues da Costa:
>
>> I'm trying to use the functions sd(x), and mean(x) in a list instead of
>> a series. From what I understood, these functions ignore NAs when used
>> in a series, but when used in a list they return NA if any missing value.
>
> Hi,
> have you actually verified that this is really what's happening? In the
> guide I see the following explanation:
>
> "The functions max, min, mean, sd, sum and var behave “horizontally”
> rather than “vertically” when their argument is a list. For instance,
> the following commands
> list Xlist = x1 x2 x3
> series m = mean(Xlist)
> produce a series m whose i-th element is the average of x 1,i ,x 2,i and
> x 3,i ; missing values, if any, are implicitly discarded."
>
> To me that sounds as if it works as you intend.
>
> cheers,
> sven
> _______________________________________________
> Gretl-users mailing list
> Gretl-users(a)lists.wfu.edu
> http://lists.wfu.edu/mailman/listinfo/gretl-users
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
10:13 a.m.
Am 05.06.2017 um 11:39 schrieb Artur Tarassow:
Let's get a concrete example:
<hansl>
set verbose off
open denmark.gdt -q
LRM[1] = NA
mean_LRM = mean(LRM)
sd_LRM = sd(LRM)
print mean_LRM sd_LRM
summary LRM --simple
list L = LRM LRY
summary L --simple
</hansl>
But I think the point is what you get after
series meanlist = mean(L)
print meanlist
-sven
12:36 p.m.
On Sat, 3 Jun 2017, Filipe Rodrigues da Costa wrote:
Dear All,
I'm trying to use the functions sd(x), and mean(x) in a list instead
of a series. From what I understood, these functions ignore NAs when
used in a series, but when used in a list they return NA if any
missing value.
Yes, that's right as a description of what gretl does, and moreover I
think it's the right thing to do. It's very standard to want
statistics for a given series ignoring any NAs in the series. But if
you want a series containing the "cross-sectional" mean of a list of
series, then in general you don't want to skip NAs. I would argue that
in that case you would in fact be calculating a different statistic at
observations where there are any NAs.
I need to compute the cross-sectional mean and standard deviation and
I
want NAs to be ignored, as they are in a series. Is there a simple way
of doing this?
There isn't a "built-in" way, but one could write a hansl function to
do it. Here's a simple example.
<hansl>
function series my_list_mean (list L)
series ret = mean(L)
series chk = missing(ret)
if nelem(L) > 1 && sum(chk) > 0
loop t=$t1..$t2 -q
if chk[t]
n_ok = 0
tot = 0
loop i=1..nelem(L) -q
if !missing(L[i][t])
tot += L[i][t]
n_ok++
endif
endloop
if n_ok > 0
ret[t] = tot / n_ok
endif
endif
endloop
endif
return ret
end function
open data4-10
list L = ENROLL CATHOL INCOME
INCOME[10] = NA
series mL = my_list_mean(L)
print L mL -o
</hansl>
Allin Cottrell
6:46 p.m.
Am 05.06.2017 um 14:36 schrieb Allin Cottrell:
On Sat, 3 Jun 2017, Filipe Rodrigues da Costa wrote:
> I need to compute the cross-sectional mean and standard deviation
and I
> want NAs to be ignored, as they are in a series. Is there a simple way
> of doing this?
There isn't a "built-in" way, but one could write a hansl function to do
it. Here's a simple example.
...
or what about using the selecting function selifc():
set skip_missings off
matrix m = {L}
matrix result = {}
loop i=1..rows(m)
result |= meanr( selifc(m, ok(m[i])) )
# and analogous with sdc( selifc(...)' )'
endloop
series mymeans = result
(untested)
cheers,
sven
6:51 p.m.
Am 05.06.2017 um 20:46 schrieb Sven Schreiber:
Am 05.06.2017 um 14:36 schrieb Allin Cottrell:
> On Sat, 3 Jun 2017, Filipe Rodrigues da Costa wrote:
>> I need to compute the cross-sectional mean and standard deviation and I
>> want NAs to be ignored, as they are in a series. Is there a simple way
>> of doing this?
>
> There isn't a "built-in" way, but one could write a hansl function to
> do it. Here's a simple example.
...
or what about using the selecting function selifc():
set skip_missings off
P.S.: AFAIK the skip_missings option only applies to the conversion
series-to-matrix. But given the problem in this thread perhaps one could
think about extending it, so that it also enables a user choice about
how to deal with missings in the cross-series operation in lists?
(But I agree the default should be that any NA propagates and produces
NAs in the result.)
thanks,
sven
8:01 p.m.
On Mon, 5 Jun 2017, Sven Schreiber wrote:
Am 05.06.2017 um 14:36 schrieb Allin Cottrell:
> On Sat, 3 Jun 2017, Filipe Rodrigues da Costa wrote:
>> I need to compute the cross-sectional mean and standard deviation and I
>> want NAs to be ignored, as they are in a series. Is there a simple way
>> of doing this?
>
> There isn't a "built-in" way, but one could write a hansl function to
do
> it. Here's a simple example.
...
or what about using the selecting function selifc():
set skip_missings off
matrix m = {L}
matrix result = {}
loop i=1..rows(m)
result |= meanr( selifc(m, ok(m[i])) )
# and analogous with sdc( selifc(...)' )'
endloop
series mymeans = result
(untested)
Something quite similar works OK (and is more concise than what I
posted). But to get the desired result the first argument to selifc()
must be a row vector, not the whole "m" matrix:
<hansl>
function series my_list_mean2 (list L)
set skip_missing off
matrix m = {L}
matrix result = {}
loop i=1..rows(m)
matrix mi = m[i,]
result |= meanr(selifc(mi, ok(mi)))
# and analogous with sdc(selifc(...)')'
endloop
series mymeans = result
return mymeans
end function
</hansl>
Allin Cottrell
9:11 p.m.
Another possibility:
<hansl>
function series my_list_mean3 (list L)
set skip_missing off
matrix m = {L}
matrix o = ok(m)
series r = sumr(o ? m : 0) ./ sumr(o)
return r
end function
</hansl>
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
10:11 a.m.
Thanks for all replies. It seems there are several alternative ways of
doing this. Here is the final look of the option I took to compute a
cross sectional mean and a cross sectional standard deviation functions:
<hansl>
function series my_cross_sectional_mean (list L)
set skip_missing off
matrix m = {L}
matrix result = {}
loop i=1..rows(m)
matrix mi = m[i,]
result |= meanr(selifc(mi, ok(mi)))
# and analogous with sdc(selifc(...)')'
endloop
series cross_sectional_mean = result
return cross_sectional_mean
end function
function series my_cross_sectional_stdev (list L)
set skip_missing off
matrix m = {L}
matrix result = {}
loop i=1..rows(m)
matrix mi = m[i,]
result |= sdc(selifc(mi, ok(mi))',sum(ok(mi))-1)'
# I'm using sum(ok(mi))-1 as degrees of freedom, to correct sdc for the
# sample standard deviation endloop
series cross_sectional_stdev = result
return cross_sectional_stdev
end function
# load example data
open data4-10
list L = ENROLL CATHOL INCOME
INCOME[10] = NA
series mL = my_cross_sectional_mean(L)
series sL = my_cross_sectional_stdev(L)
print L mL sL -o
</hansl>
I used a correction for degrees of freedom in the sdc formula. The above
code works perfectly!
Filipe
2757
days inactive
2760
days old
11 comments
5 participants
participants (5)
-
Allin Cottrell -
Artur Tarassow -
Filipe Rodrigues da Costa -
Riccardo (Jack) Lucchetti -
Sven Schreiber