I've been puzzled by real-world applications with lots of potential regressors and how the optimization of the lambda sequence turns out, especially with the BIC. Gretl seems to tell me quite often that the optimal model is one where the R2 attains 100% more or less, which I find implausible.

I'm attaching an artificial example based on the shipped fat.inp script. I changed the single-lambda setup to one with nlambda=50. The output seems to suggest to use the very liberal model with 81 non-zero coeffs (for n=80). It seems incredible that this really can be true, especially given the true model in the background. Is there some problem with the BIC calculation in these "fat" cases?

Here's some more evidence, also based on a comparison with glmnet, so for the attached script you need to have R/glmnet installed.

- Bottom line: I believe that the lambda sequence that glmnet constructs makes more sense than gretl's, at least in this case. (I have set nlambda=50 for both; AFAIK, glmnet's lambda is lambda/n in gretl's notation.)

- How do the sequences differ here? Like gretl, glmnet starts from 1.41. What's relevant then is this default setting in glmnet: lambda.min.ratio = ifelse(nobs < nvars, 0.01, 1e-04). So here it goes down only to 0.0141, while gretl's smallest lambda/n is 0.000141, two orders of magnitude less. The implied maximum (over-) fit for gretl is therefore much higher. This isn't a problem per se, if we can still find the best smaller model, but...

- ... looking at the sequence of BIC values, only glmnet gets a minimum for lambda=0.26 (index 19), resulting in a reasonable model size with Df=9. (BTW, the levels of the BICs as calculated here aren't comparable across gretl and my formula with glmnet. Only the shape and relative sizes matter.)

- Interestingly, gretl displays the same *local* minimum for lambda/n=0.26 (index 9), also df=9. BUT: for the smaller lambda/n values that aren't considered by glmnet anymore (< 0.014), gretl's BIC just plummets and so it says the super-large models should be preferred, all the way up to R2=100%.

=> Conclusion: I'm not sure why gretl's BIC values go crazy for close-to-perfect fits, but glmnet sets the minimum lambda differently (higher) for the case with more regressors than obs, and this seems like a good idea to imitate. Gretl's results for the first part of the lambda sequence are OK, though.

cheers

sven