Re: [Gretl-devel] Gretl and PUMS Data

On Thu, 25 Oct 2007, Allin Cottrell wrote: > When gretl encounters non-numeric data for a particular variable > in a CSV import it treats the values of that variable as strings, > constructs a numeric coding, and creates a "string table" that > presents the coding to the user. BUT this is done only if > non-numeric data are encountered in the first data row for the > variable in question. That is, if we read (apparently) numeric > data on rows 1 to k-1, then encounter non-numeric data on row k, > we flag an error and stop reading. > > The trouble is that some of the PUMS variables are codings, some > but not all values of which contain non-numeric characters. For > example, NAICSP, the "NAICS Industry Code", which has values > (among others) of 1133 and 113M. > > Here's a solution, perhaps not permanent if we can think of > something better: I've added a new parameter to the "set" command, > namely "codevars". You can do, for example, [...] The problem I see with this approach is that one has to know in advance which variables must be treated specially. With large datasets, you may not; the improved debugging info does help, but IMO only to an extent. A possible alternative may be the following: first, read all the data as if they were all strings. Then, with the data already in RAM, convert to numeric whenever possible. This way, you read the datafile only once, and the way stays open if we want, for instance, flag some of the variables as dummies or discrete variables straight away. What do you think?