for testing purposes I stumbled over the question of gretl's ways of running a fixed effects (FE) panel regression when the dependent variable is not time-varying. Example: <hansl> open abdata panel IND const INDOUTPT # FE per default </hansl> I would have thought that gretl either refuses to execute this, or perhaps would drop the fixed effects, or something. But it reports some -arguably strange- output wihtout explicit indication of a problem. (Is this perhaps due to the panel being unbalanced?)

Am 12.07.2019 um 16:15 schrieb Allin Cottrell: > Neither stata nor R reject this specification, but the "arguably > strange" output from gretl is indeed an artifact of sub-par numerical > precision in the unbalanced case using Cholesky decomposition. I've > switched to QR for this task and we now show something similar to > stata: > ... > coefficient std. error t-ratio p-value > ------------------------------------------------------- > const 5.12318 0.00000 NA NA > INDOUTPT 0.00000 0.00000 NA NA > > Mean dependent var 5.123181 S.D. dependent var 2.678095 Ah, very good, this looks much "better", thanks. > foreign language=stata --send-data=L > xtset unit time > > foreign language=R --send-data=L Something else: BTW, the guide mentions that --send-data is not available with Ox, but is silent for the Python case. Actually Artur and I are working (not too hard) on more tools for passing stuff to Python, but enabling --send-data would also be nice. What Python/numpy functions would you need to make this work?

On Fri, 12 Jul 2019, Allin Cottrell wrote: > On Fri, 12 Jul 2019, Sven Schreiber wrote: > >> Something else: BTW, the guide mentions that --send-data is not >> available with Ox, but is silent for the Python case. Actually Artur and >> I are working (not too hard) on more tools for passing stuff to Python, >> but enabling --send-data would also be nice. What Python/numpy functions >> would you need to make this work? > > Basically just a CSV reading function -- and presumably a target > structure that handles variable names, to make a distinction with just > sending data in matrix form. In fact, I was thinking about this some time ago: our --send-data apparatus goes back to a time when we didn't have all the data types we have now, particularly bundles. It would be really cool if we could pass bundles to languages that support some variant of associative arrays, eg R, where they call them lists, or Python, where they call them dicrtionaries (IIRC). That would give us enormnous flexibility. Of course, (a) we'd have to write import-export functions for those languages and (b) we'dd introduce some dependencies in the target languages, but I reckon that, given the mechanism we have for serialising a bundle as an xml file, that would be doable. Proof-of-concept:

Am 12.07.2019 um 18:01 schrieb Artur Tarassow: > Am 12.07.19 um 17:38 schrieb Riccardo (Jack) Lucchetti: >> >> Proof-of-concept: > > I would rather favour the json-format as an exchange protocol. Artur, I think you are in the mainstream in pushing for json to replace xml, and maybe it's a good push. However, in this specific situation we got import functions (for the specific xml produced from gretl's bwrite) for R and Python in essentially zero time. IMHO this proves that there's no need to switch to json in this particular case. Once you have your data in the foreign block (R or Python) in a native format, you can do with it anything you like, including doing json-based stuff.

> Currently, gretl can read-in json-files and converts them into a > bundle or array of bundles. Right, and perhaps json could therefore be the format of choice for the direction Python -> gretl. Do you think you could write a Python function that produces something that's readable with gretl's jsongetb?

> But there is no way function for writing a bundle's underlying > xml-structure in json-format. I think it would be doable to write a hansl function for this which could go into the "extra" package.

Gretl's doc for jsongetb says: "JSON arrays become gretl arrays, each of which holds either strings or bundles" And indeed, consider the following: <hansl> string j = sprintf("{\"vec\": [1, 2, 3]}") eval jsongetb(j).vec </hansl> which gives: Array of strings, length 3 [1] "1" [2] "2" [3] "3" So reading in data arrays would seem to need some adjustment on the gretl side.

On Fri, 12 Jul 2019, Allin Cottrell wrote: > On Fri, 12 Jul 2019, Sven Schreiber wrote: > > > Something else: BTW, the guide mentions that --send-data is not > > available with Ox, but is silent for the Python case. Actually Artur and > > I are working (not too hard) on more tools for passing stuff to Python, > > but enabling --send-data would also be nice. What Python/numpy functions > > would you need to make this work? > > Basically just a CSV reading function -- and presumably a target > structure that handles variable names, to make a distinction with just > sending data in matrix form. In fact, I was thinking about this some time ago: our --send-data apparatus goes back to a time when we didn't have all the data types we have now, particularly bundles. It would be really cool if we could pass bundles to languages that support some variant of associative arrays, eg R, where they call them lists, or Python, where they call them dicrtionaries (IIRC). That would give us enormnous flexibility. Of course, (a) we'd have to write import-export functions for those languages and (b) we'dd introduce some dependencies in the target languages, but I reckon that, given the mechanism we have for serialising a bundle as an xml file, that would be doable. Proof-of-concept: <hansl> bwrite(defbundle("x", 42, "s", "foo"), "b.xml") foreign language=R library(XML); b = xmlToList(xmlParse("b.xml")); l <- length(b); for (i in (1:l)) { type <- b[[i]]$.attrs["type"]; payload <- b[[i]]$text; print(type); switch(type, scalar={print(as.numeric(payload))}, string={print(payload)}) } end foreign </hansl>

Neither stata nor R reject this specification, but the "arguably strange" output from gretl is indeed an artifact of sub-par numerical precision in the unbalanced case using Cholesky decomposition. I've switched to QR for this task and we now show something similar to stata:

On Fri, 12 Jul 2019, Sven Schreiber wrote: Hm, there might be a slight inconsistency compared to a standard OLS specification where the dep var is constantly zero. There gretl refuses to continue, and in isolation I can see the reason for that. But in a panel context where the dep var is time invariant and you apply the within transform, then it is also identical to zero, and here gretl produces something. I guess the difference is historical, but maybe we want to rationalize things in this area. Actually, in the fixed-effects case a time-invariant dependent variable is in general not all zeros. It goes to zero on subtraction of the group means but, as in stata, we add back in the global mean. If it were truly all zeros estimation would fail as in OLS.

On Fri, 12 Jul 2019, Sven Schreiber wrote:

> OK, so it's not a pure within transformation. But still: If the dep var > is constant and non-zero, we set all other regressor coeffs to zero > (except the constant term). Fine. But if the dep var is all zeros, one > could also set all regressor coeffs to zero, including the constant, > instead of throwing an error. What makes the former case that much > different from the latter? (The estimated residuals vanish in both > cases.) True. But if a user runs a regular regression in which the dependent variable is all zeros it's surely a safe bet that he or she has made a mistake. Nothing informative can come out of it other than that conveyed by gretl's "all zeros" error message.

On Fri, 12 Jul 2019, Sven Schreiber wrote: > Am 12.07.2019 um 20:05 schrieb Allin Cottrell: > > Right. The point is more pragmatic in a programming context, as opposed > to interactive use: We came across it with the wooldridge_test_serial > package. It's just a matter of how to handle the strange corner case. As > things stand, gretl throws an error which interrupts everything. So the > authors would have to use 'catch' to continue gracefully after that > error. Since wooldridge_test_serial consists of a single, quite simple function, it seems that it could be turned into a built-in test if we think that's worthwhile.

On Sat, 13 Jul 2019, Artur Tarassow wrote: > Am 12.07.19 um 22:11 schrieb Allin Cottrell: >> On Fri, 12 Jul 2019, Sven Schreiber wrote: >> >>> Am 12.07.2019 um 20:05 schrieb Allin Cottrell: >>> >>> Right. The point is more pragmatic in a programming context, as >>> opposed >>> to interactive use: We came across it with the wooldridge_test_serial >>> package. It's just a matter of how to handle the strange corner >>> case. As >>> things stand, gretl throws an error which interrupts everything. So >>> the >>> authors would have to use 'catch' to continue gracefully after that >>> error. >> >> Since wooldridge_test_serial consists of a single, quite simple >> function, it seems that it could be turned into a built-in test if >> we think that's worthwhile. > > Good idea, Allin! In case it does not mean too much work for you. That's now done in git. "modtest --autocorr" (or the "Autocorrelation" menu item under Tests in the model window) gives the Wooldridge test, provided the estimator is fixed or random effects and the panel has a time-series length of at least 3.

On Sun, 14 Jul 2019, Artur Tarassow wrote: > Am 13.07.19 um 23:12 schrieb Allin Cottrell: >> >> That's now done in git. "modtest --autocorr" (or the >> "Autocorrelation" menu item under Tests in the model window) >> gives the Wooldridge test, provided the estimator is fixed or >> random effects and the panel has a time-series length of at least >> 3. >> > Hi Allin, thanks for implementing this test. I've realized that > your results differ from the package's one. I think the reasons > are: > > 1) The wooldridge_test_serial() function does currently not work > correctly in case an intercept is passed as an regressor as it is > automatically omitted due to the "diff" command -- this is not > correctly controlled in the package. You can try this and check > the print-out: > > <hansl> > open abdata.gdt > list X = const n k > matrix M = wooldridge_test_serial(ys,X) > print M > </hansl> Yes, I noticed that. The built-in version preserves the constant in the regression list. More on this point below. > 2) I've corrected issue (1) such that the point estimates of the > "First-Differenced equation" and the auxiliary regression on the > residuals are equal to the one gretl produces. However, the > standard errors are different -- I don't know why. Ah, I was using the internal lsq() function for the auxiliary regressions, and the robust option invoked a White-style HCCME, not the panel-clustered variant. I've now amended the built-in code so that it gives clustered standard errors. > 3) The degrees of freedom used for computing the test statistics > also differ. The wooldridge_test_serial() function, for instance, > applies the $df accessor which accesses the number of > cross-sectional units after the ols command given a panel data > set. That's also to do with clustering or not. In the cluster-robust case the df equals (-1 plus) the number of clusters (i.e. cross-sectional units), not the number of observations. I didn't notice that because I was working with the Wooldridge chapter 10 example, where T = 3 and the number of obs in the residual autoregression equals the number of units. (With T > 3 this regression does not boil down to a cross section and the two df variants diverge.) Back to the issue of the constant. I notice that "xtserial" (Wooldridge test for Stata) does not include a constant in the FD equation. The FD specification includes a constant only if it "appears" via differencing of a time trend, or is otherwise present in the span of the differenced regressors. Maybe that's right. In his chapter 10 example Wooldridge includes a constant in his FD specification but that doesn't settle the matter: his use of const and d89 is equivalent to omitting the constant but including the differences of his d88 and d89 dummies (as xtserial would). Allin _______________________________________________ Gretl-devel mailing list -- gretl-devel(a)gretlml.univpm.it To unsubscribe send an email to gretl-devel-leave(a)gretlml.univpm.it Website: https://gretlml.univpm.it/postorius/lists/gretl-devel.gretlml.univpm.it/

On Sat, 13 Jul 2019, Artur Tarassow wrote:

That's now done in git. "modtest --autocorr" (or the "Autocorrelation" menu item under Tests in the model window) gives the Wooldridge test, provided the estimator is fixed or random effects and the panel has a time-series length of at least 3. > However, in another private chat, Sven has posed the question whether > Wooldridge's test is really only valid for RE-/FE-type of panel > models. Once this is clarified it may be applicable for a larger class > of panel models. OK, I'll stay tuned on that point.

On Sun, 14 Jul 2019, Sven Schreiber wrote: > I think it's clear that FE and RE are the most important use cases, > probably covering 95-99% of all needs here. In principle, however, I > guess that at least two more variants could be easily covered: First > there's 'panel ... --pooled', because it's just a special case when > there are no individual effects. I've now added (via modtest --autocorr) the serial correlation test that Wooldrige recommends for pooled OLS (see pp. 176-7 of his panel book). It's based on an auxiliary pooled regression in which the lagged residual is added to the original specification; the test is just a (robust) t-test on the lag.

Am 15.07.19 um 19:18 schrieb Sven Schreiber: > Am 15.07.2019 um 18:13 schrieb Allin Cottrell: >> On Sun, 14 Jul 2019, Sven Schreiber wrote: >> >>> I think it's clear that FE and RE are the most important use cases, >>> probably covering 95-99% of all needs here. In principle, however, I >>> guess that at least two more variants could be easily covered: First >>> there's 'panel ... --pooled', because it's just a special case when >>> there are no individual effects. >> I've now added (via modtest --autocorr) the serial correlation test >> that Wooldrige recommends for pooled OLS (see pp. 176-7 of his panel >> book). It's based on an auxiliary pooled regression in which the >> lagged residual is added to the original specification; the test is >> just a (robust) t-test on the lag. Thanks, Allin! @Giuseppe: I guess, this makes the wooldridge_test_serial package obsolete, right? Or does it cover some 'special' case?

I agree with Artur: Allin's built-in commands are much better and versatile and there is no reason to keep the wooldridge_test_serial alive.

=2019d that the package isn't needed anymore for them.