On Fri, 22 Sep 2017, Allin Cottrell wrote:
On Thu, 21 Sep 2017, Schaff, Frederik wrote:
> unfortunately this works only for 2 but not 3 or more records... And I do
> not understand why.
Frederik is referring to my script to do his data concatenation job using the
(recently tweaked) "append" command; see
http://lists.wfu.edu/pipermail/gretl-users/2017-September/012690.html
for details.
I see what the problem is. First, there was a residual issue with the
"append" command, which I have now fixed in git. But beyond that, I was
assuming a greater regularity in Frederik's data files than actually obtains.
The following comment will mean little
to anyone who hasn't tried working with Frederik's data, but here goes...
My assumption was that the per-observation files
stat_0-499_1.tsv
stat_0-499_2.tsv
stat_0-499_3.tsv
(and other such tuples) contained observations (1,2,3) on a common set of
variables. Mostly that does seem to be the case -- and if it were always the
case my "append" recipe should work -- but it's not always the case. In
particular the file stat_0-499_3.tsv contains three variable identifiers that
are not found in any other file, namely
Attendance_L_cv_I1
Attendance_L_Skew_I1
Attendance_L_Kurt_I1
This discrepancy means the "append" (a fairly simple-minded command) will not
do the job [...]
Ah, but now I understand the source of the discrepancy, and it
should be fixable, if not already fixed, by Frederik. The data files
that he posted originally (the "1" and "2" files) contained the
invalid identifiers:
Attendance_L-cv_I1
Attendance_L-Skew_I1
Attendance_L-Kurt_I1
These are invalid because "-" is an operator (minus) from gretl's
point of view and you can't stick operators into identifiers. So
gretl "auto-corrected" these names to
Attendance_Lcv_I1
Attendance_LSkew_I1
Attendance_LKurt_I1
(just dropping the offending character, as explained in the doc for
gretl's fixname() function). Meanwhile, however, Frederik himself
corrected these names to
Attendance_L_cv_I1
Attendance_L_Skew_I1
Attendance_L_Kurt_I1
(replacing '-' with '_') in the "3" file that he sent more
recently.
So I presume that with current gretl git/snapshot my "append" recipe
should work after all, provided one is working with a consistent set
of Frederik's data files (either all auto-corrected in gretl's
manner, or all manually corrected in an alternative but consistent
manner).
Allin