[Gretl-users] Re: Reading large csv and sorting data set -- a comparison with 2 python libs

On Mon, 14 Dec 2020, Artur Tarassow wrote: [...] > Am 13.12.20 um 15:35 schrieb Allin Cottrell: > >> Sorting a dataset: This was not optimized for a huge number of >> observations. We were allocating more temporary storage than >> strictly necessary and moreover, at some points, calling malloc() >> per observation when it was possible to substitute a single call to >> get a big chunk of memory. Neither of these points were much of an >> issue with a normal-size dataset but they became a serious problem >> in the huge case. That's now addressed in git. > > As I already wrote you privately: This change is a boost as sorting > time got reduced from 14 to 7.5 seconds. Thanks for this! > > By the way, does this increased speed in sorting also affect the > aggregate() function? Right now it's specific to the case of sorting an entire dataset, but it would be worth taking a look at the aggregate case too.

>> gdtb is not designed for a dataset like this. I guess you'd want an >> uncompressed pure-binary format for best read and write speed and >> minimal memory consumption. > > I guess you're right but how can I store data as a pure-binary format > without any compression? I can't find anything on this in the help text. Well, no. That's not a format that gretl has offered to date. But see below. > By the way, I stored the csv file as a gdt file via "store > <FILENAME>.gdt". To my surprise, the gdt file is 843 MB large and > hence almost 2.5 times larger than the csv. That shouldn't be suprising if you've ever looked "inside" a gdt file. The CSV file just contains the data. The structured XML file contains per-observation tags. With a single data series (mostly smallish integers), the tags occupy more bytes than the data.

> Also, trying to open the file consumes massive RAM here on my Ubunut > 20.04 machine: RAM increases from 1GB before starting to read the > file to 15GB (and actually slightly more but the swap file already > starts to increase). Libxml2 is having to allocate a ton of memory to parse the entire document. XML is just not the way to go for this volume of data. There's an experiment ("proof of concept") in git which might be of interest. I've enabled reading and writing "pure binary" data files, with ".gbin" suffix. Here are my initial test scripts. Read the original data and write gbin: <hansl> set stopwatch # users.csv has first column named "obs" open users.csv printf "load time %gs\n", $stopwatch summary n --simple set stopwatch dataset sortby n printf "sort time %gs\n", $stopwatch summary n --simple set stopwatch store users.gbin printf "gbin store time %gs\n", $stopwatch </hansl> The times I'm seeing are: read csv:   9.95s sort by n:  6.09s write gbin: 1.40s Then restart and read the gbin: <hansl> set stopwatch open users.gbin printf "gbin open time %gs\n", $stopwatch summary n --simple </hansl> I see an open time of 1.76s. As I said, it's at proof-of-concept stage. String-valued series are not handled and only minimal metadata are conveyed (variable names, observation markers if present, basic time-series info).