Re: [Gretl-devel] memory with many discrete variables in large (panel) dataset

On Wed, 1 Jan 2014, Riccardo (Jack) Lucchetti wrote: > On Wed, 1 Jan 2014, Sven Schreiber wrote: > >> Nevertheless, given the fact that many of the variables in there are >> discrete integer-valued which could be saved as a signed single byte, >> and that (I think) I know that gretl stores every variable as >> double-precision 8-byte numbers, there are huge potential memory savings >> and speed improvements. Maybe that's something to think about for the >> longer term. > > This is an issue Allin and I have been discussing since the Toruń > conference. Eventually we dropped the idea, because to properly > support series which contain anything but doubles we'd have to rewrite > pretty much everything internally (however, examining the issue did > lead us to a substantial rewrite of the internals, which made things > more efficient and streamlined --- see the commits made around the end > of July 2011 if you're curious). I realise I wasn't quite clear: the reason why I said "we've been discussing" is that at least myself haven't totally given up the idea. However, it was put on hold indefinitely because it wasn't obvious that the advantages would outweigh the massive rewrite effort that would be necessary.