Re: [Gretl-devel] memory with many discrete variables in large (panel) dataset

On Wed, 8 Jan 2014, Sven Schreiber wrote: >> 2) Writing binary data >> >> Tweaks to our writing of data in text form are be useful, but >> there's no question that if you want raw speed you're better using >> C's fwrite and fread to zap big swathes of bytes from RAM to disk or >> vice versa. I've implemented a --binary option to "store" that >> causes gretl to write out an XML .gdt file containing the metadata >> plus a binary .bdt file containing doubles. > > Hm, that sounds as if now a .gdt file could indicate either a > traditional standalone file or a new metadata file, which are quite > different things, no? A new suffix would seem in order. (.mgdt?) I wouldn't mind something like that, especially because gdt files have always been self-contained. With the new "binary" thing, instead, the dataset is spread between 2 files, with the gdt file containing the metadata only. The problem is, it would be difficult to tell a 'self-contained' gdt file from a 'metadata-only' gdt file from the outside, which could be a problem in some cases.

> Yes I'm a big fan of compression level 1 -- why not make that the default? Good idea.