Hi again, as you will notice, I am currently working on a real-life panel dataset, and so here's another panel problem, with example; the following doesn't work for me ("no obs would be left"): <hansl> open abdata.gdt smpl IND == 3 --restrict # leaves 89 obs smpl unit == min(unit) --restrict # lowest unit value of the 89 obs is 6 </hansl>

Here's another problem that may be related, because a common theme is the combination of panel data and subsampling. But now it's about the pxsum() function. Here's a minimal example script: <hansl> nulldata 12 setobs 4 1:1 --stacked-time-series series hey = normal() hey[1:3] = NA genr unit smpl unit < 3 --restrict series check1 = pxsum(hey) series check2 = pxsum(misszero(hey)) series check3 = pxsum(ok(hey)) </hansl> 'check1' will be correct, with missings in positions 1:3 and 2:3. (There has been a feature request on the tracker by myself for a long time calling for a behavior of pxsum that ignores missings, but that has never been done AFAIK and thus is a different issue.) But 'check2' and 'check3' will be full of missings / completely empty at least within the current subsample, and that is clearly wrong IMO. Instead check2 and check3 should have no missings at all. Hope it's easy to find and fix the bug.

That's now found and fixed. The p* functions for panel statistics, such as pxsum, were not confining themselves to the current sample range. That wasn't a completely motiveless crime, since I originally thought of them as providing values pertaining to the entire panel dataset. Nonetheless, I agree that failing to respect the sample range produces counter-intuitive results and is out of line with other functions.

as you will notice, I am currently working on a real-life panel dataset, and so here's another panel problem, with example; the following doesn't work for me ("no obs would be left"): <hansl> open abdata.gdt smpl IND == 3 --restrict # leaves 89 obs smpl unit == min(unit) --restrict # lowest unit value of the 89 obs is 6 </hansl> My guess is that gretl evaluates min(unit) not as the lowest in-sample value (after applying the first restriction), but instead looks for the global minimum (which is 1). Then the two restrictions cannot hold simultaneously.

But I think that all gretl functions that take a series argument always work on the currently defined sample, right? If so, then this looks like a bug.

Am 06.12.2015 um 03:42 schrieb Allin Cottrell: > On Fri, 4 Dec 2015, Sven Schreiber wrote: > >> as you will notice, I am currently working on a real-life panel dataset, >> and so here's another panel problem, with example; the following doesn't >> work for me ("no obs would be left"): >> >> <hansl> >> open abdata.gdt >> smpl IND == 3 --restrict # leaves 89 obs >> smpl unit == min(unit) --restrict # lowest unit value of the 89 obs is 6 >> </hansl> >> >> My guess is that gretl evaluates min(unit) not as the lowest in-sample >> value (after applying the first restriction), but instead looks for the >> global minimum (which is 1). Then the two restrictions cannot hold >> simultaneously. > > That's right. This is required if > > smpl <whatever> --restrict --replace > > is going to work. Yes, but I did not have the --replace option in there !? The guide (in section 5.3) says that "the second restriction augments the first". Admittedly that does not pin down exactly the scope of the second restriction, but I find this very surprising. To put it differently, the expression "min(unit)" is currently evaluated differently in identical circumstances (with respect to the active sample and the contents of the series), if you compare my example with your solution. > >> But I think that all gretl functions that take a series argument always >> work on the currently defined sample, right? If so, then this looks like >> a bug. > > I wouldn't say so. ?? Is this a "no" to the first sentence or just to the second?

Partly unrelated: surely priority number one is to fix buggy behaviour, that's a given. However, having said this, I've found that for complex scripting involving panel datasets the combined action of the aggregate() and replace() function allows you to do amazing things.

With aggregate() by the "time" variable I'm now running into the problem that gretl tells me time is not discrete. And in the GUI attributes I cannot set it to be discrete.

I have to say it's quite frustrating, I'm getting nowhere except uncovering bugs and other limitations. One of the nice things with gretl is that data handling is usually nice and efficient. With panel data my impression is different. And now think about someone without my level of gretl experience...

>> I have to say it's quite frustrating, I'm getting nowhere except >> uncovering bugs and other limitations. One of the nice things with gretl >> is that data handling is usually nice and efficient. With panel data my >> impression is different. And now think about someone without my level of >> gretl experience... > > But you're not just any other gretl user. You're a gretl developer, and > this is the devel-list. Bug eradication is one of our jobs. Today I didn't want to be a developer, I just wanted to use gretl. The reason I was posting to the devel list is because of what I found.

On Sun, 6 Dec 2015, Sven Schreiber wrote:

>> But you're not just any other gretl user. You're a gretl developer, and >> this is the devel-list. Bug eradication is one of our jobs. > > Today I didn't want to be a developer, I just wanted to use gretl. The > reason I was posting to the devel list is because of what I found. Handling panel datasets is objectively difficult. With all the limitations and bugs you exposed, I still believe we're _leaps and bounds_ above R in that respect (which I consider as our closest competitor). "But Stata", I hear you say. OK, that's what they charge you money for: they do their debugging behind closed doors. We do ours in public. That's the "O" in FLOSS.

On Sun, 6 Dec 2015, Sven Schreiber wrote: > With aggregate() by the "time" variable I'm now running into the problem > that gretl tells me time is not discrete. And in the GUI attributes I > cannot set it to be discrete. That was fixed in git and snapshots a few days ago. The GUI was withholding the discrete option when a variable had "too many" distinct values. But that shouldn't apply if the variable has only integer values, and it doesn't any more.

Ah, for future reference: there's a much more efficient way to do the same: instead of $i_med = replace(unit, M[,1], M[,j++]) you can use $i_med2 = pexpand(M[,j++]) which is of course more general but much faster.