7:08 a.m.
While playing with large datasets, I found that the "omit" command wouldn't
work for a particular model: I got a "Data error" message. Some investigation
led me to discover (I didn't know before) that, in order for the omit command
to work, you need to have the same number of observations for both the
unrestricted and the restricted model. Same goes for "add".
The reason for this is that the test that gets computed by these commands
involves comparing the two models, which is ok. However, for large datasets,
this is a very serious limit: when you have many variables and thousands of
observations the likelihood of having missing values in some of the
added/omitted variables is very high.
Now, this can be bypassed via "restrict", which does a Wald-style test, but if
you want to exclude dozens of variables at the same time (think of industry
dummies, for instance), it's not the handiest of tools.
Therefore, my proposal (which probably involves some work, but not much): how
about appending a --wald option to "omit", by which we carry out the test by
the "restrict" machine instead of the default?
Riccardo "Jack" Lucchetti
Dipartimento di Economia
Facoltà di Economia "G. Fuà"
Ancona
8:10 a.m.
On Thu, August 31, 2006 13:08, Riccardo Jack Lucchetti"" wrote:
Therefore, my proposal (which probably involves some work, but not
much): how
about appending a --wald option to "omit", by which we carry out the test by
the "restrict" machine instead of the default?
Attached you'll find a proof of concept: a diff file to apply to current CVS
which adds the --wald option to the omit command and a sample script. The neat
thing, embodied in the very idea of the Wald test, is that we don't need to
estimate the restricted model at all, so
1) the CPU time for computing the test statistic is very small (this could be
a problem with large datasets);
2) the Wald approach is applicable whenever missing values get in the way of
comparing the two models.
Allin, if you like the idea, please consider that my implementation is still
very rough (especially wrt the model[0]<->model[1] stuff), and you'll have to
sort out the mess I'll have certainly made; to make things worse, the code in
the patch is almost comment-free. I wouldn't mind at all if this went in after
release.
Riccardo "Jack" Lucchetti
Dipartimento di Economia
Facoltà di Economia "G. Fuà"
Ancona
8:20 a.m.
On Fri, September 1, 2006 14:10, Riccardo Jack Lucchetti"" wrote:
Attached you'll find a proof of concept: a diff file to apply to
current CVS
which adds the --wald option to the omit command and a sample script.
Sorry, the sample script was missing. Here it is.
Riccardo "Jack" Lucchetti
Dipartimento di Economia
Facoltà di Economia "G. Fuà"
Ancona
1:07 p.m.
On Fri, 1 Sep 2006, Riccardo Jack Lucchetti wrote:
Attached you'll find a proof of concept: a diff file to apply
to current CVS which adds the --wald option to the omit
command and a sample script...
Thanks, this is good. I've implemented a version of the
suggestion, but I used the existing wald_test() code in
compare.c, with slight modifications for better handling
of errors.
Allin.
8:30 p.m.
New subject: The "omit" command once again
Thanks to Jack for shaking things up on this point.
In current CVS, the English help file contains a more accurate
description than hitherto of what the "omit" command actually
does. It's a bit convoluted. I'll try to summarize more
critically here.
(1) "Traditionally", the default behaviour of "omit" was to
estimate the restricted model (that is, the source model minus
the independent variables selected for omission), and to push
this restricted model onto the stack as the "last model
estimated". In the process, if the source model was estimated
using OLS, we calculated the F-test for the joint significance
of the omitted variables, using the restricted and unrestricted
SSRs and their respective degrees of freedom,
(2) Some time ago, it occurred to me that this was a bit weak
with regard to models estimated via methods other than OLS (no
test statistic was provided); furthermore, it was problematic
with regard to models estimated via OLS but calling for "robust
standard errors" (an innovation relative to old-style gretl).
(3) I therefore modified the behavior of "omit" so that
(i) if the original model used OLS with "robust standard
errors", we formulated a Wald-type statistic using the robust
covariance matrix from the original model, in F-form, and
(ii) if the original estimator was something other than OLS, we
used a Wald test (chi-square form) based on the covariance
matrix for the original model.
Modifications (i) and (ii) enabled us to present a test
statistic for the joint significance of the omitted variables in
cases where we'd previously skipped the reporting of such a
statistic. However, I maintained the old behaviour of
estimating (and "returning") the restricted model, even if that
model was not used in computing the test statistic for the
omitted variables.
Jack is now pointing out that in some cases we really don't want
to base our test statistic on explicit estimation of the
restricted model -- even if that method is valid "in principle",
we'd rather just use the Wald method on the original covariance
matrix. I agree.
Here's the question I'm working towards: Do you think we'd be
better off if we changed the behaviour of the "omit" command so
that it calculates and reports a test statistic -- possibly by
means of estimating the restricted model, depending on the
original estimator and relevant option switches -- but never
replaces the "current model" with a restricted model?
In practical scripting terms:
ols 1 0 2 3 4 5
omit 4 5
omit 4 5
Currently, the third line above is an error: you can't omit vars
4 and 5 because they're already omitted: by this time the
"current model" has spec "ols 1 0 2 3".
In the possible new-style behaviour, line 3 would not be an
error, it would simply repeat the test done on line 2.
In current CVS, line 3 would _not_ be an error if we used the
"--wald" option flag, which suppresses estimation of the
restricted model. That is,
ols 1 0 2 3 4 5
omit 4 5 --wald
omit 4 5 --wald
would be valid, if repititious.
Allin.
5:33 a.m.
New subject: The "omit" command once again
On Sat, September 2, 2006 02:30, Allin Cottrell wrote:
Here's the question I'm working towards: Do you think
we'd be
better off if we changed the behaviour of the "omit" command so
that it calculates and reports a test statistic -- possibly by
means of estimating the restricted model, depending on the
original estimator and relevant option switches -- but never
replaces the "current model" with a restricted model?
Thanks for the recap. Here's my take on this.
The "omit" command, IMO, should be considered mainly a testing tool; there are
two motivations for this: one is computational efficiency; the other is more
methodological. In my mind, the action of restricting an existing model (the
estimation technique is immaterial in this context) is composed of two
logically consecutive steps: FIRST you test if your restriction is at odds
with data, and IF NOT you want see what the restricted model looks like. In
other words, there is no use for a restricted model whose validity has already
been established as dubious.
Besides, in many cases we may want to perform a variable omission test hoping
that the null is rejected: this is exactly why you breathe that sigh of relief
when the t-statistic on your precious regressor that you really care about is
over 2. In those cases, you don't want to see the restricted model, you just
want to make sure regressors aren't redundant.
It so happens that the most widely used estimation tool in econometrics (ols)
lets you merge the two steps into one; in fact, when you do a variable
omission F-test the traditional way, you effectively reverse the order of the
two steps. This comes from the fact that, under ideal conditions (no missing
values in the dataset, homoskedasticity, no serial correlation etc.), the
actual calculations necessary for the two steps are practically the same.
However, I see this as a by-product of the particular way ols works; the
logical priority order stays unchanged.
So, my proposal is to reverse the logic of the omit command. By default, a
test statistic should be what you get; that said, it would be silly not to be
able to estimate the restricted model in one go. So, I wouldn't mind somehing
like
ols 1 0 2 3 4
omit 3 4 # just show the test statistic
omit 3 4 --new # estimate the restricted model and show you the test
Of course, "--new" stands for "estimate a new model in the process",
which is
not a particularly transparent way to put it, but I couldn't think of anything
better at this moment; needless to say, a similar change would also involve
the "add", "addto" and "omitfrom" commands.
I do realise that this would be a rather drastic change, so I wouldn't tear my
hair out in despair if we kept things as they are now for backward
compatibility.
Ideas?
Riccardo "Jack" Lucchetti
Dipartimento di Economia
Facoltà di Economia "G. Fuà"
Ancona
6608
days inactive
6610
days old
5 comments
2 participants
participants (2)
-
Allin Cottrell -
Riccardo Jack Lucchetti