Re: [Gretl-devel] opening large spss file kills the app without error message
On Sat, 16 Mar 2019, Boris Demeshev wrote:
Thanks a lot for the support!!!
There are 4 files (individuals spss, individuals stata, households
spss, households stata). For the moment I can not open Individuals
files (neither stata nor spss). Will check with current build and
again next week! :)
I've been working with the Stata individuals (longitudinal) file,
USER_RLMS-HSE_IND_1994_2017_v2_eng.dta
There was a bug (a typo, really) in our dta reader which prevented
it from reading the "map" of offsets properly when some elements
were bigger than 32-bit integers. That's now fixed in git and
snapshots and I can open the file OK on a machine with 16 GB of RAM.
It's a struggle on an older machine with 8 GB. The uncompressed
Stata file is about 3.8 GB, and the gretl in-memory representation
of the full dataset takes up almost 7 GB. That's because all data
are stored as doubles in gretl -- we don't have a "byte" data type
as in Stata, which permits economical encoding of dummy variables.
However, if you plan to use only a relatively small subset of the
2723 series, you can open the dta file on a machine with plenty of
RAM and save a subset in gretl native format. The reduced file will
then be readable on most machines.
Allin