1:25 p.m.
Hi all,
it looks as if regls (Lasso...) isn't checking for missings that happen
somewhere in the middle of the data. Implicitly this is acknowledged by
comments in the example code snippets in the regls help document
("without missing values..." etc.). I just saw some internal data errors
happening because of that. I guess it would be good if regls checked for
such missings and then fails more gracefully.
This is with gretl 2025a.
thanks
sven
2:09 p.m.
Am 15.04.2025 um 15:25 schrieb Sven Schreiber:
it looks as if regls (Lasso...) isn't checking for missings that
happen somewhere in the middle of the data. Implicitly this is
acknowledged by comments in the example code snippets in the regls
help document ("without missing values..." etc.). I just saw some
internal data errors happening because of that. I guess it would be
good if regls checked for such missings and then fails more gracefully.
Another regls (regls_multiprint) problem, although unrelated to the one
above (I think; after missings have been taken care of):
lfrac R^2 sum(|b|) df criterion
1.00000 0.0000 0.000000 0
Uneindeutiger Matrixindex # ---> ambiguous index
*** Fehler in Schleife in Funktion regls_multiprint
printf " %#10g\n", rb.crit[i]
Perhaps this is a corner case? R2=0? Does somebody have a successful
regls_multiprint invocation?
thanks
sven
11:43 a.m.
On Tue, Apr 15, 2025 at 9:25 AM Sven Schreiber
<sven.schreiber(a)fu-berlin.de> wrote:
it looks as if regls (Lasso...) isn't checking for missings that happen
somewhere in the middle of the data. Implicitly this is acknowledged by
comments in the example code snippets in the regls help document
("without missing values..." etc.). I just saw some internal data errors
happening because of that. I guess it would be good if regls checked for
such missings and then fails more gracefully.
Please provide a minimal example. The regls addon includes code to
detect and avoid missing values, as follows:
<hansl>
# handle possible missing values
list All = depvar indeps
series okcheck = ok(All)
if sum(okcheck) < $nobs
set matrix_mask okcheck
endif
</hansl>
This seems to work in most cases so I think there must be something
unusual going on.
Allin
12:09 p.m.
On Tue, Apr 15, 2025 at 10:09 AM Sven Schreiber
<sven.schreiber(a)fu-berlin.de> wrote:
Another regls (regls_multiprint) problem [...]
lfrac R^2 sum(|b|) df criterion
1.00000 0.0000 0.000000 0
Uneindeutiger Matrixindex # ---> ambiguous index
*** Fehler in Schleife in Funktion regls_multiprint
> printf " %#10g\n", rb.crit[i]
Perhaps this is a corner case? R2=0? Does somebody have a successful
regls_multiprint invocation?
R^2 is zero by construction when lfrac = 1. Successful invocations of
multiprint are provided in the example file multiprint.inp. Please
give the context in which this error arose.
Allin
12:16 p.m.
Am 16.04.2025 um 14:09 schrieb Cottrell, Allin:
On Tue, Apr 15, 2025 at 10:09 AM Sven Schreiber
<sven.schreiber(a)fu-berlin.de> wrote:
> Another regls (regls_multiprint) problem [...]
>
> lfrac R^2 sum(|b|) df criterion
> 1.00000 0.0000 0.000000 0
> Uneindeutiger Matrixindex # ---> ambiguous index
> *** Fehler in Schleife in Funktion regls_multiprint
> > printf " %#10g\n", rb.crit[i]
>
> Perhaps this is a corner case? R2=0? Does somebody have a successful
> regls_multiprint invocation?
R^2 is zero by construction when lfrac = 1. Successful invocations of
multiprint are provided in the example file multiprint.inp. Please
give the context in which this error arose.
Thanks, I'm attaching an example adapted from multiprint.inp which
provokes the error here for me. It seems to be related to cross-validation.
cheers
sven
12:29 p.m.
Am 16.04.2025 um 13:43 schrieb Cottrell, Allin:
Please provide a minimal example. The regls addon includes code to
detect and avoid missing values, as follows:
...
This seems to work in most cases so I think there must be something
unusual going on.
Yeah, right now I cannot replicate it anymore. I'll keep an eye open and
will post more details when/if it happens next time.
cheers
sven
2:05 p.m.
On Wed, Apr 16, 2025 at 8:29 AM Sven Schreiber
<sven.schreiber(a)fu-berlin.de> wrote:
Am 16.04.2025 um 13:43 schrieb Cottrell, Allin:
> Please provide a minimal example. The regls addon includes code to
> detect and avoid missing values, as follows:
> ...
> This seems to work in most cases so I think there must be something
> unusual going on.
Yeah, right now I cannot replicate it anymore. I'll keep an eye open and
will post more details when/if it happens next time.
Were you by any chance calling mregls() rather than regls()? In mregls
we were assuming that the matrix input would not contain any
non-finite values; I've now added a check on that.
Allin
2:07 p.m.
Am 16.04.2025 um 16:05 schrieb Cottrell, Allin:
On Wed, Apr 16, 2025 at 8:29 AM Sven Schreiber
<sven.schreiber(a)fu-berlin.de> wrote:
> Am 16.04.2025 um 13:43 schrieb Cottrell, Allin:
>> Please provide a minimal example. The regls addon includes code to
>> detect and avoid missing values, as follows:
>> ...
>> This seems to work in most cases so I think there must be something
>> unusual going on.
> Yeah, right now I cannot replicate it anymore. I'll keep an eye open and
> will post more details when/if it happens next time.
Were you by any chance calling mregls() rather than regls()?
No, not this time.
In mregls
we were assuming that the matrix input would not contain any
non-finite values; I've now added a check on that.
Cannot hurt, I guess.
thanks
sven
2:08 p.m.
On Wed, Apr 16, 2025 at 8:16 AM Sven Schreiber
<sven.schreiber(a)fu-berlin.de> wrote:
Am 16.04.2025 um 14:09 schrieb Cottrell, Allin:
> On Tue, Apr 15, 2025 at 10:09 AM Sven Schreiber
> <sven.schreiber(a)fu-berlin.de> wrote:
>> Another regls (regls_multiprint) problem [...]
>>
>> lfrac R^2 sum(|b|) df criterion
>> 1.00000 0.0000 0.000000 0
>> Uneindeutiger Matrixindex # ---> ambiguous index
>> *** Fehler in Schleife in Funktion regls_multiprint
>> > printf " %#10g\n", rb.crit[i]
>>
>> Perhaps this is a corner case? R2=0? Does somebody have a successful
>> regls_multiprint invocation?
> R^2 is zero by construction when lfrac = 1. Successful invocations of
> multiprint are provided in the example file multiprint.inp. Please
> give the context in which this error arose.
Thanks, I'm attaching an example adapted from multiprint.inp which
provokes the error here for me. It seems to be related to cross-validation.
Thanks for the example. The problem was specific to the case where
multiprint was called after cross validation. It's now fixed in git.
Allin
3:33 p.m.
New subject: regls (...)
OK, here's another regls question, and again nothing to do with missings
anymore -- I thought I re-interpret this thread as being regls-related
in general today.
I've been puzzled by real-world applications with lots of potential
regressors and how the optimization of the lambda sequence turns out,
especially with the BIC. Gretl seems to tell me quite often that the
optimal model is one where the R2 attains 100% more or less, which I
find implausible.
I'm attaching an artificial example based on the shipped fat.inp script.
I changed the single-lambda setup to one with nlambda=50. The output
seems to suggest to use the very liberal model with 81 non-zero coeffs
(for n=80). It seems incredible that this really can be true, especially
given the true model in the background. Is there some problem with the
BIC calculation in these "fat" cases?
thanks
sven
3:15 p.m.
New subject: regls (...)
On Wed, Apr 16, 2025 at 11:33 AM Sven Schreiber
<sven.schreiber(a)fu-berlin.de> wrote:
I've been puzzled by real-world applications with lots of potential
regressors and how the optimization of the lambda sequence turns out,
especially with the BIC. Gretl seems to tell me quite often that the
optimal model is one where the R2 attains 100% more or less, which I
find implausible.
I'm attaching an artificial example based on the shipped fat.inp script.
I changed the single-lambda setup to one with nlambda=50. The output
seems to suggest to use the very liberal model with 81 non-zero coeffs
(for n=80). It seems incredible that this really can be true, especially
given the true model in the background. Is there some problem with the
BIC calculation in these "fat" cases?
Yes, it seems clear that BIC is not an effective criterion in the fat
case. The trouble is that as a saturated model is approached the SSR
will continue to decline somewhat, increasing the log-likelihood,
while the penalty 'k' in the BIC formula (which in regls lasso is the
number of non-zero coefficients) tends to stabilize at around the
number of observations. The net effect is that the BIC continues to
"improve" regardless as lambda shrinks. Use of cross validation, on
the other hand, produces sensible results.
I notice that glmnet does two relevant things in the fat case: (1) it
sets by default a relatively large value (0.01) for the smallest
lambda fraction when the user just specifies a number of lambda
values, and (2) it automatically terminates exploration of small
lambda when the R^2 reaches 0.991 or so. We might do something
similar. Glmnet doesn't produce BIC values (though I see that Scikit
Learn does); if we continue to show them we should probably issue a
warning in the fat case.
Allin
3:18 p.m.
New subject: regls (...)
Am 16.04.2025 um 17:33 schrieb Sven Schreiber:
I've been puzzled by real-world applications with lots of
potential
regressors and how the optimization of the lambda sequence turns out,
especially with the BIC. Gretl seems to tell me quite often that the
optimal model is one where the R2 attains 100% more or less, which I
find implausible.
I'm attaching an artificial example based on the shipped fat.inp
script. I changed the single-lambda setup to one with nlambda=50. The
output seems to suggest to use the very liberal model with 81 non-zero
coeffs (for n=80). It seems incredible that this really can be true,
especially given the true model in the background. Is there some
problem with the BIC calculation in these "fat" cases?
Here's some more evidence, also based on a comparison with glmnet, so
for the attached script you need to have R/glmnet installed.
- Bottom line: I believe that the lambda sequence that glmnet constructs
makes more sense than gretl's, at least in this case. (I have set
nlambda=50 for both; AFAIK, glmnet's lambda is lambda/n in gretl's
notation.)
- How do the sequences differ here? Like gretl, glmnet starts from 1.41.
What's relevant then is this default setting in glmnet: lambda.min.ratio
= ifelse(nobs < nvars, 0.01, 1e-04). So here it goes down only to
0.0141, while gretl's smallest lambda/n is 0.000141, two orders of
magnitude less. The implied maximum (over-) fit for gretl is therefore
much higher. This isn't a problem per se, if we can still find the best
smaller model, but...
- ... looking at the sequence of BIC values, only glmnet gets a minimum
for lambda=0.26 (index 19), resulting in a reasonable model size with
Df=9. (BTW, the levels of the BICs as calculated here aren't comparable
across gretl and my formula with glmnet. Only the shape and relative
sizes matter.)
- Interestingly, gretl displays the same *local* minimum for
lambda/n=0.26 (index 9), also df=9. BUT: for the smaller lambda/n values
that aren't considered by glmnet anymore (< 0.014), gretl's BIC just
plummets and so it says the super-large models should be preferred, all
the way up to R2=100%.
=> Conclusion: I'm not sure why gretl's BIC values go crazy for
close-to-perfect fits, but glmnet sets the minimum lambda differently
(higher) for the case with more regressors than obs, and this seems like
a good idea to imitate. Gretl's results for the first part of the lambda
sequence are OK, though.
cheers
sven
3:25 p.m.
New subject: regls (...)
Am 17.04.2025 um 17:15 schrieb Cottrell, Allin:
Yes, it seems clear that BIC is not an effective criterion in the
fat
case. The trouble is that as a saturated model is approached the SSR
will continue to decline somewhat, increasing the log-likelihood,
while the penalty 'k' in the BIC formula (which in regls lasso is the
number of non-zero coefficients) tends to stabilize at around the
number of observations.
Ah, our messages overlapped, see my other email. Your very
intuitive
reasoning explains my open question about gretl's BIC results there.
The net effect is that the BIC continues to
"improve" regardless as lambda shrinks. Use of cross validation, on
the other hand, produces sensible results.
I notice that glmnet does two relevant things in the fat case: (1) it
sets by default a relatively large value (0.01) for the smallest
lambda fraction when the user just specifies a number of lambda
values, and (2) it automatically terminates exploration of small
lambda when the R^2 reaches 0.991 or so.
Yes, I read that, too, although I'm
not sure if the auto-termination
already applied in my example, since I got the full 50 requested lambda
values.
We might do something
similar. Glmnet doesn't produce BIC values (though I see that Scikit
Learn does); if we continue to show them we should probably issue a
warning in the fat case.
Yes; since under your reasoning the minimal BIC will tend to be a corner
solution at the right edge, it might be worth reporting the local
minimum, too. Although finding a local minimum is of course slightly
more involved than a global one.
thanks
sven
7
days inactive
9
days old
12 comments
2 participants
participants (2)
-
Cottrell, Allin -
Sven Schreiber