[Gretl-devel] Re: problem with large gdtb panel datafile and index variables

On Fri, 14 Jul 2023, Sven Schreiber wrote: > Am 14.07.2023 um 17:34 schrieb Cottrell, Allin: >> >> >> I tried mocking up a fairly hefty panel (though no doubt a good deal >> smaller than yours) and I got a pause of several seconds between steps >> 3 and 4. Is that the point at which gretl became unresponsive in your >> case? > > Yes! Good, so we're on the same page. ... * But also in the GUI we'd like to produce the most explicit possible error message, so the (duplicated) check involves an extra, expensive step. We check explicitly for the pathology whereby for one or more of the (unit, period) pairs there are two or more observations in the dataset. Errors of this sort will be caught by "setobs", but in the GUI we go an extra step so we can provide details: for example, "The combination of unit 12 and period 5 occurs at both observation 3445 and observation 7981".

As things stand, the mechanism for doing this is of roughly quadratic complexity in the total number of observations. Possibilities are: skip the extra code for N*T > some sane limit, or, if possible, find a much more efficient implementation.