On Wed, 1 Jan 2014, Sven Schreiber wrote: This is an issue Allin and I have been discussing since the Toruń conference. Eventually we dropped the idea, because to properly support series which contain anything but doubles we'd have to rewrite pretty much everything internally (however, examining the issue did lead us to a substantial rewrite of the internals, which made things more efficient and streamlined --- see the commits made around the end of July 2011 if you're curious). ------------------------------------------------------- Riccardo (Jack) Lucchetti Dipartimento di Scienze Economiche e Sociali (DiSES) Università Politecnica delle Marche (formerly known as Università di Ancona) r.lucchetti(a)univpm.it http://www2.econ.univpm.it/servizi/hpp/lucchetti -------------------------------------------------------

Am 01.01.2014 15:14, schrieb Riccardo (Jack) Lucchetti: It is not surprising that the whole issue is problematic, and nobody wants a disruptive rewrite just because of that. But actually, the RAM memory usage itself is not such a big deal IMO, given today's typical RAM capacities. Instead, what made me think about this at all was the noticeable delay when opening and especially saving the workfile. So maybe it would be enough to speed up the gzipping, perhaps by trading off the compressed size. For example, instead of the 42:3.5 relation between the Stata and gretl files, getting instead a 10MB gretl file would still be much smaller than the Stata competition, but the threefold increase of the file size may allow the gzip to be sufficiently faster? thanks, sven

On Wed, 1 Jan 2014, Sven Schreiber wrote: I'd like to experiment with this. Can you give a little more detail on the characteristics of the data file? That is (roughly) how many observations? And how many variables? And what sort of ratio of quantitative variables to small-integer coded variables? I've tried generating a random dataset with 10000 observations on 850 variables, 50 of them normal and the remaining 800 binary. On disk This occupies 26MB uncompressed and 7MB with maximal gzip compression. Reading the gzipped version takes a little longer but in neither case is the delay very noticeable. So I'd like to know which dimension(s) to increase to make the gzipped load time problematic. Allin

On Thu, 2 Jan 2014, Sven Schreiber wrote: Gretl requires that a panel dataset have equal T_i for all units i, even if this means padding with NAs (which is done automatically when you use "setobs" with the --panel-vars option). So if the dataset is very "holey", it's not too surprising that the full panel could be a lot bigger than the minimal undated, unpadded dataset. However, it seems that we shouldn't necessarily save all the padding rows when writing a gdt file; all we need to do is save all the rows that actually contain some data, plus enough information to reconstitute the panel on reading. There's now an experiment to this effect in CVS (it has been tested, but more testing would be good). Here's the deal: * When saving a panel dataset in native format we check the total size of the panel. If this exceeds 10MB we then check what fraction of the data rows are pure padding, and if this exceeds 30% we go into "skip-padding" mode. (These parameters are of course somewhat arbitary and could be tweaked.) * In skip-padding mode we skip the pure padding rows (duh!) but we record the (1-based) unit index and time index for each row we write, these appearing as two extra series at the end of the dataset; we also record the fact that we're skipping. * On re-reading the gdt file, gretl recognizes that a panel should be reconstituted, and it uses the unit and time info to do that. There's room for debate over what the extra, automatically added series should be called. Right now they're called "unit__" and "time__" (with trailing double underscores). These names seem unlikely to collide with user-specified ones. Even if they do, that won't matter if the file is read by current gretl, since if we get the skip-padding flag we know to use the last two series regardless of what they're called (and delete them after using them). The naming of these series is intended to ease sharing of data with older versions of gretl: if need be, a "skip-padding" panel dataset could be reconstituted manually by open skip-padding.gdt setobs unit__ time__ --panel-vars In this case a name collision could be a problem -- in the (unlikely) event that the dataset already contains series of these names, and they do not constitute a valid basis for --panel-vars. Related: The --gzipped option to the "store" command now has an optional compression-level parameter. This should be an integer in the range 0-9, with 0 indicating no compression and 9 maximal. If no parameter is given, the default level is 6 (the zlib default), which in most cases seems to offer a reasonable balance between speed and compressed size. The GUI File save dialog for native gdt files now also has a compression level selector. Allin

On Mon, 6 Jan 2014, Sven Schreiber wrote: I've now done some testing for speed of "skip-padding" and actually the improvement is not all that great. It turns out that with a big and severely unbalanced panel it takes quite a while to count the padding rows. I'm showing my results below. In all cases the simulated datasets had 1200 variables and about 70 percent of the rows were filled with NAs to represent panel padding; the cases differ by NT (the number of rows in the balanced panel) and the compression level used. Overall I'm seeing a slight gain in (compress + write) speed, and a more substantial but still not dramatic gain in (read + decompress) speed. While the size of the gdt file on disk is smaller with skip-padding, when compression is enabled the difference is not as great as one might imagine. Evidently zlib does a pretty good job of shrinking the padding rows to next-to-nothing. case 1 (NT = 10000, gzip = 0) original with skip ratio store(sec) 2.97 2.80 0.943 open(sec) 0.76 0.45 0.591 disk(Kb) 37224 12836 0.345 case 2 (NT = 60000, gzip = 6) original with skip ratio store(sec) 22.77 21.27 0.934 open(sec) 5.10 2.82 0.552 disk(Kb) 24161 22415 0.928 case 3 (NT = 60000, gzip = 1) original with skip ratio store(sec) 18.55 16.90 0.911 open(sec) 5.11 2.77 0.542 disk(Kb) 28746 26126 0.909 Allin

On Tue, 7 Jan 2014, Sven Schreiber wrote: Well, not quite. In my testing on a panel dataset with the general characteristics we've been talking about, skip-padding (other things) equal always shaved about 10% off the write time and about 40% off the read time. However, those results have now been overtaken by some new thinking. Sorry, this is a bit long but here we go. The basic gdt design (gzipped XML with a fairly simple but extensible DTD) is IM0 nice and clean, and it can in principle handle data of any size. But it's certainly slower than some alternatives, and with very big datasets speed can become a serious bother. With this in mind, I've been working on two things in CVS. The first relates to the way we print numbers in a gdt file and the second concerns the possibility of using a binary format. 1) Writing data as text When we're writing out numerical data in text form, as in an XML file, we need to preserve precision. Double-precision (64-bit) floating point values are generally good for about 15 or 16 significant digits. To be on the safe side, we have for some time now written data values to 17 significant digits. To be precise, we use the printf format "%.17g", which does not write trailing zeros. So if you open up a gdt file in a text editor you'll see that some values are actually printed with 17 digits (e.g. generated logs and pseudo-random values), while at the other end of the spectrum dummies are justed printed as 1s and 0s. The question arises, what about primary data that are not just dummies (or other small-integer codings)? Primary economic data are most unlikely to carry more than 6 significant digits (maybe 9 at the outside, but that's probably spurious) and often carry fewer. However, if you print such data using %.17g you get nasty artifacts arising from the limited precision of doubles: a value that has been published as 8.98 (and therefore should really be recorded in the gdt file as just that) appears as 8.9800000000000004, one that has been published as 217.004 appears as 217.00399999999999, and so on. Such artifacts increase the size and decrease the compressibility of the XML file. So we try to get rid of them. When we're getting ready to save as gdt series whose values are in the "middling" size range occupied by most primary data, we try to figure out a format that will represent the series with full precision but cut out the artifacts. This works OK, but it takes time, and with very big data the time becomes non-trivial. What's new here is that I've worked out a substantially faster way of determining the appropriate the format specification for each series. In my tests this cuts several seconds off big data writes. This improvement is independent of the compression level and the skip-padding status. 2) Writing binary data Tweaks to our writing of data in text form are be useful, but there's no question that if you want raw speed you're better using C's fwrite and fread to zap big swathes of bytes from RAM to disk or vice versa. I've implemented a --binary option to "store" that causes gretl to write out an XML .gdt file containing the metadata plus a binary .bdt file containing doubles. If compression is specified it applies to both files. The bdt file is written in host endianness; the reading function should take care of conversion if need be (but that's not tested yet since I don't have access to any big-endian machine.) The speed-up via this approach is dramatic: 600 MB or so can be written in 0.54 seconds and read back in 0.17 seconds. I've put a table with my current results at http://ricardo.ecn.wfu.edu/~cottrell/tmp/binzip.html This shows the relationship between binary vs text, compression vs none, and skip padding vs not. As you'll see, if speed is the only concern (and disk space not an issue), a straight binary write/read with no compression or skipping of padding wins the race. On the other hand you can get quite nice performance using skip-padding and compression level 1: write in 2.4 seconds, read in 1.2 seconds, and use only 4 percent of the maximal disk space. Allin

Am 08.01.2014 04:31, schrieb Allin Cottrell: ... Yes the results are impressive! I'm just wondering whether it would also help in this context to use the information on which variables are officially "discrete". Most of them (not all) will be integer-valued for example. Also, in principle the data are now changed with respect to the old format I guess; hopefully just within the precision error margin of doubles, but this should probably be tested -- or did you already? BTW, I have always wondered how gretl actually determines the true/false value of something like "indicator == 1" given that the series are stored as floating-point values and the corresponding representation problems of integers... Hm, that sounds as if now a .gdt file could indicate either a traditional standalone file or a new metadata file, which are quite different things, no? A new suffix would seem in order. (.mgdt?) Yes I'm a big fan of compression level 1 -- why not make that the default? Again, impressive! thanks, sven

Am 08.01.2014 09:52, schrieb Riccardo (Jack) Lucchetti: Right, could have thought of that myself. Actually I've just learned that quite a lot of integers can be represented exactly. Yep, that's exactly what I meant. -sven

On Wed, 8 Jan 2014, Sven Schreiber wrote: Good idea. Even if they're not integer-valued, they certainly should not require 17 digits. The artifacts that I mentioned disappear if you use the printf format %.15g, and this could safely be applied to series marked as discrete without any need for elaborate testing. I did. We need 17 signifcant digits if we're to reproduce exactly results obtained using (e.g.) logs and random numbers (by reproduce I mean: run a regression, save data, reopen data, run regression again.) But we still use 17 digits if that's required -- that is, if printing to 15 digits doesn't leave trailing zeros. [Personally, I think logs and random numbers should be generated by script not saved in a data file, but anyway.] Allin