Some news regarding gretl's "join" command (importation of data with
lots of options). These points are in the current documentation for
"join" in the User's Guide, but I thought it would be worth
explicitly drawing them to people's attention.
1) I've mentioned this before but only in passing: besides "CSV"
(delimited text) files you can now join from gretl-native gdt or
gdtb (binary) files.
2) More recent: you can now pull multiple series from the source
file in one command.
I'll expand on the second point. When we first wrote "join" we were
wrestling with a lot of complexity (key-matching, filtering,
aggregation) and we simplified matters by stipulating that only a
single series could be operated on at a time. Now that the join code
has stabilized, we've found it feasible to support "batch"
importation of series. This is subject to two limitations:
1) When importing multiple series, the --data option (which permits
renaming of a single series on import) is not available. You have to
accept the names of series as they appear in the source data file
(or as "fixed up" by gretl, if need be).
2) You only get one set of key-matching, filtering and aggregation
options; these options are applied uniformly to all series
specified in a single command. So if you want to import several
series but with different keys, filters or aggregation methods,
you still need separate instances of the "join" command.
How do you ask for multiple series? You just replace the second
(series-name) argument to "join" with either (a) several series
names, separated by spaces, or (b) the name of an array-of-strings
variable that holds the names of the series you want.
My motivation for setting this up is that this semester I've been
helping some students construct datasets from the PUMS (Public Use
Microdata Sample) made available by the US Census Bureau. These are
BIG files (e.g. the person datafile for California alone is >
300MB). So if you want data from all 50 US states plus DC, and
especially if you want household-level data too, we're talking quite
a major data processing exercise. I've found that with multiple
imports in "join" it doesn't take much longer to import 6 or 7
series at a time than it does to import a single series, meaning
that we get a very noticeable speed-up of the process.
Show replies by date