7:06 p.m.
Please post to the "Gretl-users" mailing list the following question:
I have easily weighted data in SPSS in the past, and now I would like to weight European
Social Survey (ESS) data in the GRETL software package, if possible, but I don't know
how. I would be most grateful if you advised me.
The syntax I use in SPSS for this purpose is as follows: WEIGHT BY analysisweight .
Best Regards,Evangelos Bekiaris PhD candidate University of Peloponnese
4:25 p.m.
On Wed, 20 Jul 2022, Μπεκιάρης Ευάγγελος wrote:
I have easily weighted data in SPSS in the past, and now I would like
to
weight European Social Survey (ESS) data in the GRETL software package,
if possible, but I don't know how. I would be most grateful if you
advised me. The syntax I use in SPSS for this purpose is as follows:
WEIGHT BY analysisweight .
You should look at the documentation for the "wls" command.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
7:03 p.m.
The problem is that I won't do simple OLS estimations but 2SLS...
My aim is to perform a general weighting, specifically post stratification weighting, in
order to conduct estimations with instrumental variables. This is easy in SPSS using the
simple syntax:
WEIGHT BY pspwght
or other variables ESS provide, like dweight, pweight, anweight or a combination of them.
Then the weighted can be analyzed with almost any technique such 2SLS, OLS, Multilevel
modelling etc.
I cannot do something similar in Gretl, which has a more powerful routine for 2SLS
relative to SPSS....
9:50 a.m.
On Thu, 21 Jul 2022, elfsog(a)yahoo.gr wrote:
The problem is that I won't do simple OLS estimations but 2SLS...
[...]
I cannot do something similar in Gretl, which has a more powerful
routine for 2SLS relative to SPSS....
It wouldn't be difficult to write a Hansl script to achieve this: once the
cross-product matrices are computed, the rest is simple algebra.
However, from a purely statistical point of view, the benefits of
weighting are not at all obvious. In fact, in some cases, weighting could
be even detrimental for efficiency. See eg
https://www.oecd.org/skills/piaac/events/Item_10_Jeffrey_Wooldridge.pdf
This IMO explains why weighted linear models are not particularly popular
in econometrics, outside certain special cases.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
10:27 a.m.
Am 22.07.2022 um 11:50 schrieb Riccardo (Jack) Lucchetti:
On Thu, 21 Jul 2022, elfsog(a)yahoo.gr wrote:
> The problem is that I won't do simple OLS estimations but 2SLS...
[...]
> I cannot do something similar in Gretl, which has a more powerful
> routine for 2SLS relative to SPSS....
It wouldn't be difficult to write a Hansl script to achieve this: once
the cross-product matrices are computed, the rest is simple algebra.
Isn't it even simpler than that? I mean, if you have a known weighting
factor you multiply that with the relevant series to get a weighted
result, and from there you can go on as you usually would. Of course
there's a manual pre-step involved and so it isn't quite as handy as if
there were a specialized setting available, OK. But no need for special
scripting, or am I missing something?
However, from a purely statistical point of view, the benefits of
weighting are not at all obvious. In fact, in some cases, weighting
could be even detrimental for efficiency. See eg
https://www.oecd.org/skills/piaac/events/Item_10_Jeffrey_Wooldridge.pdf
True.
cheers
sven
10:42 a.m.
On Fri, 22 Jul 2022, Sven Schreiber wrote:
Am 22.07.2022 um 11:50 schrieb Riccardo (Jack) Lucchetti:
> On Thu, 21 Jul 2022, elfsog(a)yahoo.gr wrote:
>
>> The problem is that I won't do simple OLS estimations but 2SLS...
>
> [...]
>
>> I cannot do something similar in Gretl, which has a more powerful
>> routine for 2SLS relative to SPSS....
>
> It wouldn't be difficult to write a Hansl script to achieve this: once
> the cross-product matrices are computed, the rest is simple algebra.
Isn't it even simpler than that? I mean, if you have a known weighting
factor you multiply that with the relevant series to get a weighted
result, and from there you can go on as you usually would. Of course
there's a manual pre-step involved and so it isn't quite as handy as if
there were a specialized setting available, OK. But no need for special
scripting, or am I missing something?
Well, in the context of tsls things may be a little more complicated than
that, especially when it comes to computing valid standard errors. But it
may be conjectured that doing weighted IV "by hand" by using wls instead
of ols in the first and second stage would deliver identical estimates to
a full-fledged weighted IV routine. (As I said, computing valid standard
errors might be a little more intircate).
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
3:10 p.m.
Am 22.07.2022 um 12:42 schrieb Riccardo (Jack) Lucchetti:
On Fri, 22 Jul 2022, Sven Schreiber wrote: > >> Am
22.07.2022 um 11:50 schrieb Riccardo (Jack) Lucchetti: >>> On
Thu, 21 Jul
2022, elfsog(a)yahoo.gr wrote: >>> >>>> The problem is that I
won't do simple OLS estimations but >>>> 2SLS... >>> >>>
[...] >>> >>>>
I cannot do something similar in Gretl, which has a more >>>> powerful
routine for 2SLS relative to SPSS.... >>> >>> It wouldn't be
difficult
to write a Hansl script to achieve this: >>> once the cross-product
matrices are computed, the rest is simple >>> algebra. >> >>
Isn't it
even simpler than that? I mean, if you have a known >> weighting factor
you multiply that with the relevant series to get >> a weighted result,
and from there you can go on as you usually >> would. Of course there's
a manual pre-step involved and so it isn't >> quite as handy as if there
were a specialized setting available, >> OK. But no need for special
scripting, or am I missing something? > > Well, in the context of tsls
things may be a little more complicated > than that, especially when it
comes to computing valid standard > errors. But it may be conjectured
that doing weighted IV "by hand" by > using wls instead of ols in the
first and second stage would deliver > identical estimates to a
full-fledged weighted IV routine. (As I > said, computing valid standard
errors might be a little more
intircate).
OK, so I don't know much about this post-stratification stuff that
Evangelos mentions.
But in generic terms of weighted IV estimation, I'm not convinced that
this is so complicated. For example, Stata's doc for ivregress just
laconically says, (paraphrased) "if you have weights just squeeze in
this D matrix and you'll be fine".
(See here: https://www.stata.com/manuals13/rivregress.pdf, p.12 -
specifically, "X_1'X_1, X'X, X'y, y'y, Z'Z, Z'X, and Z'y
are replaced
with [always a D in the middle]", where D is a diagonal matrix with
normalized weights.)
So isn't this the same as multiplying everything (X_1, X_2, Y, y; in
Stata's notation) with the square roots of the normalized weights and
then doing standard IV? Just like WLS amounts to multiplying everything
with the root of the weight and then running OLS.
Yes, R2 has to be treated specially, but the rest?
cheers
sven
4:20 p.m.
On Fri, 22 Jul 2022, Sven Schreiber wrote:
But in generic terms of weighted IV estimation, I'm not convinced
that
this is so complicated. For example, Stata's doc for ivregress just
laconically says, (paraphrased) "if you have weights just squeeze in
this D matrix and you'll be fine".
(See here: https://www.stata.com/manuals13/rivregress.pdf, p.12 -
specifically, "X_1'X_1, X'X, X'y, y'y, Z'Z, Z'X, and Z'y
are replaced
with [always a D in the middle]", where D is a diagonal matrix with
normalized weights.)
So isn't this the same as multiplying everything (X_1, X_2, Y, y; in
Stata's notation) with the square roots of the normalized weights and
then doing standard IV? Just like WLS amounts to multiplying everything
with the root of the weight and then running OLS.
Yes, R2 has to be treated specially, but the rest?
Since you mentioned Stata, I thought I'd try a little numerical experiment
on my earlier conjecture, and, surprisingly enough, it appears that I was
right: if you run 2-step estimation substituing wls for ols, the
coefficient estimates are exactly spot-on. The standard error, however are
not. Here's my example script:
<hansl>
set verbose off
function void wtsls(series wt, series depvar, list reg, list inst)
scalar k = nelem(reg)
list exo = reg && inst
list endo = x - exo
list U = null
series resc_w = wt/mean(wt)
loop foreach i endo
wls wt $i inst --quiet
u_$i = $uhat
U += u_$i
endloop
wls resc_w depvar reg U --quiet --robust
b = $coeff[1:k]
v = $vcv[1:k, 1:k]
cs = b ~ sqrt(diag(v))
pn = varnames(reg)
modprint cs pn
end function
nulldata 128
y = normal()
x = normal()
z1 = normal()
z2 = normal()
wt = exp(normal())
list X = const x
list Z = const z1 z2
/*
tsls y X ; Z
foreign language=stata --send-data
ivreg y (x=z1 z2)
end foreign
*/
wtsls(wt, y, X, Z)
foreign language=stata --send-data
ivreg y (x=z1 z2) [pweight=wt]
end foreign
</hansl>
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
7:08 p.m.
Am 22.07.2022 um 18:20 schrieb Riccardo (Jack) Lucchetti:
On Fri, 22 Jul 2022, Sven Schreiber wrote:
> (See here: https://www.stata.com/manuals13/rivregress.pdf, p.12 -
> specifically, "X_1'X_1, X'X, X'y, y'y, Z'Z, Z'X, and
Z'y are replaced
> with [always a D in the middle]", where D is a diagonal matrix with
> normalized weights.)
>
> So isn't this the same as multiplying everything (X_1, X_2, Y, y; in
> Stata's notation) with the square roots of the normalized weights and
> then doing standard IV? Just like WLS amounts to multiplying everything
> with the root of the weight and then running OLS.
>
> Yes, R2 has to be treated specially, but the rest?
Since you mentioned Stata, I thought I'd try a little numerical
experiment on my earlier conjecture, and, surprisingly enough, it
appears that I was right: if you run 2-step estimation substituing wls
for ols, the coefficient estimates are exactly spot-on.
I believe this is basically
one part of the point I'm trying to make.
The standard error, however are not.
And this strikes me as the standard result when you run IV explicitly in
two stages, that you would need to adjust the second stage in that respect.
Here's in hansl form what I have in mind:
<hansl-add-to-Jacks-script>
# just weight everything
series w_factor = sqrt(wt / mean(wt))
loop foreach i const y x z1 z2
series $i_w = w_factor * $i
endloop
list Xw = const_w x_w
list Zw = const_w z1_w z2_w
tsls y_w Xw ; Zw --no-tests
</hansl-add-to-Jacks-script>
The coefficients agree exactly with those from your function.
Now, I haven't cross-checked the standard errors there with Stata or
anything. I tend to think they should be right, but I'm not certain.
Maybe you could have another look.
thanks
sven
4:36 p.m.
Am 22.07.2022 um 21:08 schrieb Sven Schreiber:
The coefficients agree exactly with those from your function.
Now, I haven't cross-checked the standard errors there with Stata or
anything. I tend to think they should be right, but I'm not certain.
Maybe you could have another look.
I remembered that I have Stata on this machine, so I cross-checked with
the help of your nice foreign-invoking script. If i add the --robust
option to my weighted tsls variant, then I get exactly Stata's output,
including the standard errors for inference (well, except the R2, as
expected).
So to sum up, here's an example how to do weighted IV regression in gretl:
<hansl>
nulldata 128
# artificial input data
y = normal()
x = normal()
z1 = normal()
z2 = normal()
# artificial weighting
wt = exp(normal())
# root normalized weights
series w_factor = sqrt(wt / mean(wt))
# multiply to each variable (including the constant!)
loop foreach i const y x z1 z2
series $i_w = w_factor * $i
endloop
list Xw = const_w x_w
list Zw = const_w z1_w z2_w
# run IV on the weighted data
tsls y_w Xw ; Zw --no-tests --robust
</hansl>
Again, disregard the R2 and related statistics in the output.
In principle this could be put in a function like Jack did with the
2-stage wls approach, then could be packaged up to also treat stuff like
the variable names properly and so on. Plus perhaps calculate a correct
R2. But the econometrics as such is in the few lines above.
cheers
sven
8:44 p.m.
I am not familiar with the survey methodology used in the ESS. I
understand that it is based on a complex survey methodology. As such using
the weights in standard wls will not give correct results. The survey
package in R can give correct estimates. I think that stata can also do
this. Lumley (2010), Complex surveys - A guide to analysis using R.
Wiley. Lohr (2010), Sampling Design and Analysis 2nd edition, has a more
introductory treatment.
I have commented here because I have seen an analysis go very wrong because
of this problem. Please excuse me if my comment is out of order.
John C Frain
3 Aranleigh Park
Rathfarnham
Dublin 14
Ireland
www.tcd.ie/Economics/staff/frainj/home.html
https://jcfrain.wordpress.com/
https://jcfraincv19.wordpress.com/
mailto:frainj@tcd.ie
mailto:frainj@gmail.com
On Thu, 21 Jul 2022 at 17:25, Riccardo (Jack) Lucchetti <
p002264(a)staff.univpm.it> wrote:
On Wed, 20 Jul 2022, Μπεκιάρης Ευάγγελος wrote:
> I have easily weighted data in SPSS in the past, and now I would like to
> weight European Social Survey (ESS) data in the GRETL software package,
> if possible, but I don't know how. I would be most grateful if you
> advised me. The syntax I use in SPSS for this purpose is as follows:
> WEIGHT BY analysisweight .
You should look at the documentation for the "wls" command.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------_______________________________________________
Gretl-users mailing list -- gretl-users(a)gretlml.univpm.it
To unsubscribe send an email to gretl-users-leave(a)gretlml.univpm.it
Website:
https://gretlml.univpm.it/postorius/lists/gretl-users.gretlml.univpm.it/
9:01 p.m.
I think there's no gretl command similar to SPSS "weight by " in the sense
that it can not be activated by default for the active dataset (as you can
do in SPSS though the "weight observations" menu.)
Instead you need to specifically indicate a weighting variable whenever
needed (as for example in the case of wls) or applied (as for example in
the 'summary' command).
Best,
Artur
Le jeu. 21 juil. 2022, 21:44, John C Frain <frainj(a)gmail.com> a écrit :
I am not familiar with the survey methodology used in the ESS. I
understand that it is based on a complex survey methodology. As such using
the weights in standard wls will not give correct results. The survey
package in R can give correct estimates. I think that stata can also do
this. Lumley (2010), Complex surveys - A guide to analysis using R.
Wiley. Lohr (2010), Sampling Design and Analysis 2nd edition, has a more
introductory treatment.
I have commented here because I have seen an analysis go very wrong
because of this problem. Please excuse me if my comment is out of order.
John C Frain
3 Aranleigh Park
Rathfarnham
Dublin 14
Ireland
www.tcd.ie/Economics/staff/frainj/home.html
https://jcfrain.wordpress.com/
https://jcfraincv19.wordpress.com/
mailto:frainj@tcd.ie
mailto:frainj@gmail.com
On Thu, 21 Jul 2022 at 17:25, Riccardo (Jack) Lucchetti <
p002264(a)staff.univpm.it> wrote:
> On Wed, 20 Jul 2022, Μπεκιάρης Ευάγγελος wrote:
>
> > I have easily weighted data in SPSS in the past, and now I would like
> to
> > weight European Social Survey (ESS) data in the GRETL software package,
> > if possible, but I don't know how. I would be most grateful if you
> > advised me. The syntax I use in SPSS for this purpose is as follows:
> > WEIGHT BY analysisweight .
>
> You should look at the documentation for the "wls" command.
>
> -------------------------------------------------------
> Riccardo (Jack) Lucchetti
> Dipartimento di Scienze Economiche e Sociali (DiSES)
>
> Università Politecnica delle Marche
> (formerly known as Università di Ancona)
>
> r.lucchetti(a)univpm.it
> http://www2.econ.univpm.it/servizi/hpp/lucchetti
>
>
-------------------------------------------------------_______________________________________________
> Gretl-users mailing list -- gretl-users(a)gretlml.univpm.it
> To unsubscribe send an email to gretl-users-leave(a)gretlml.univpm.it
> Website:
> https://gretlml.univpm.it/postorius/lists/gretl-users.gretlml.univpm.it/
>
_______________________________________________
Gretl-users mailing list -- gretl-users(a)gretlml.univpm.it
To unsubscribe send an email to gretl-users-leave(a)gretlml.univpm.it
Website:
https://gretlml.univpm.it/postorius/lists/gretl-users.gretlml.univpm.it/
8:01 a.m.
Am 21.07.2022 um 23:01 schrieb Artur Bala:
I think there's no gretl command similar to SPSS "weight by
" in the > sense that it can not be activated by default for the active dataset
(as you can do in SPSS though the "weight observations" menu.) >
Instead you need to specifically indicate a weighting variable >
whenever needed (as for example in the case of wls) or applied (as > for
example in the 'summary' command). >
I think you're right, Artur. Thanks also for reminding us/me of the
--weight option of the 'summary' command. So I think what we have in
gretl for weighing stuff is:
- 'summary ... --weight' (but this weighs the observations, not the
variables, I think)
- the functions wmean(), wvar(), wsd()
- the 'wls' command, as mentioned before
Maybe there's more.
thanks
sven
8:14 a.m.
On Fri, 22 Jul 2022, Sven Schreiber wrote:
Am 21.07.2022 um 23:01 schrieb Artur Bala:
> I think there's no gretl command similar to SPSS "weight by " in the
>
> sense that it can not be activated by default for the active dataset
> (as you can do in SPSS though the "weight observations" menu.) >
Instead you need to specifically indicate a weighting variable >
whenever needed (as for example in the case of wls) or applied (as > for
example in the 'summary' command). >
I think you're right, Artur. Thanks also for reminding us/me of the
--weight option of the 'summary' command. So I think what we have in
gretl for weighing stuff is:
- 'summary ... --weight' (but this weighs the observations, not the
variables, I think)
- the functions wmean(), wvar(), wsd()
- the 'wls' command, as mentioned before
Maybe there's more.
There's the "wstats" package for weighted equivalents of the
"freq" and
"xtab" commands.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
883
days inactive
886
days old
14 comments
6 participants
participants (6)
-
Artur Bala -
elfsog＠yahoo.gr -
John C Frain -
Riccardo (Jack) Lucchetti -
Sven Schreiber -
Μπεκιάρης Ευάγγελος