3:12 p.m.
Hi,
another minor glitch: I had a large Stata dta file and imported it into
gretl - this actually worked flawlessly via dragging-and-dropping (on
Windows). Then I went on to save it; in the save dialog gretl inserted
the extension "gdt" automatically, and when I changed the format to the
binary gdtb type in the drop-down box at the bottom, in the end I had a
file ending in .gdt.gdtb.
Obviously very easy to correct manually in various ways along the steps
taken, but maybe now we want to shake out even minor stuff like this.
thanks
sven
3:28 p.m.
Am 11.01.2021 um 16:12 schrieb Sven Schreiber:
Hi,
another minor glitch: I had a large Stata dta file and imported it into
gretl - this actually worked flawlessly via dragging-and-dropping (on
Windows). Then I went on to save it; in the save dialog gretl inserted
the extension "gdt" automatically, and when I changed the format to the
binary gdtb type in the drop-down box at the bottom, in the end I had a
file ending in .gdt.gdtb.
Obviously very easy to correct manually in various ways along the steps
taken, but maybe now we want to shake out even minor stuff like this.
Hm, I'm observing something strange with this conversion to gdtb, and
using the choice of "new missing codes".
[BTW I don't remember exactly, does the compression level actually apply
to the gdtb format? I get a file size of 25900KB no matter whether I
choose 0 or 1 for that level.]
Opening this 26MB file causes a very noticeable delay of several
seconds. OK, the PC is not brand new, but hey... This is a panel dataset
with 26 time periods and roughly 18000 units. I somehow seem to remember
that the gdtb format was introduced because of speed, and indeed it felt
much faster back then (but not with this very same dataset, admittedly).
thanks
sven
6:38 p.m.
On Mon, 11 Jan 2021, Sven Schreiber wrote:
Am 11.01.2021 um 16:12 schrieb Sven Schreiber:
> Hi,
> another minor glitch: I had a large Stata dta file and imported it into
> gretl - this actually worked flawlessly via dragging-and-dropping (on
> Windows). Then I went on to save it; in the save dialog gretl inserted
> the extension "gdt" automatically, and when I changed the format to the
> binary gdtb type in the drop-down box at the bottom, in the end I had a
> file ending in .gdt.gdtb.
> Obviously very easy to correct manually in various ways along the steps
> taken, but maybe now we want to shake out even minor stuff like this.
Hm, I'm observing something strange with this conversion to gdtb, and
using the choice of "new missing codes".
Please try the test under "data-io" at
https://sourceforge.net/p/gretl/workspace/ci/master/tree/
Here's the output I get from running MB26.sh, which creates a
dataset of (somewhat greater than) the 26 MB you mention, and saves
then loads it in various ways:
plain gdt : save time 1.649s, size 72745837 bytes
plain gdt : load time 0.950s
trad. gdtb : save time 0.889s, size 25103074 bytes
trad. gdtb : load time 0.148s
gdtb gzipped=6: save time 1.204s, size 25007970 bytes
gdtb gzipped=6: load time 0.140s
purebin : save time 0.022s, size 29121024 bytes
purebin : load time 0.021s
There's not much to be got out of gzip compression because the data
are random normal. "Traditional" gdtb is faster for both writing and
reading than plain gdt. The new "purebin" is of course much faster
still.
Allin
10:03 p.m.
Am 11.01.2021 um 19:38 schrieb Allin Cottrell:
On Mon, 11 Jan 2021, Sven Schreiber wrote:
> Am 11.01.2021 um 16:12 schrieb Sven Schreiber:
>> Hi,
>> another minor glitch: I had a large Stata dta file and imported it into
>> gretl - this actually worked flawlessly via dragging-and-dropping (on
>> Windows). Then I went on to save it; in the save dialog gretl inserted
>> the extension "gdt" automatically, and when I changed the format to
the
>> binary gdtb type in the drop-down box at the bottom, in the end I had a
>> file ending in .gdt.gdtb.
>> Obviously very easy to correct manually in various ways along the steps
>> taken, but maybe now we want to shake out even minor stuff like this.
>
> Hm, I'm observing something strange with this conversion to gdtb, and
> using the choice of "new missing codes".
Please try the test under "data-io" at
https://sourceforge.net/p/gretl/workspace/ci/master/tree/
Sorry, observing this on Windows here I just did a much cruder
comparison: Loading the same dataset from gdtb (new I think) and from
gdt. BTW gdt in this heavily unbalanced case is smaller at only 18MB
compared to 26MB before. I get:
- 6.6s for gdtb
- 14.7s for gdt
So yes, the binary stuff still helps a lot. It was just my expectation
that 26MB somehow "should" be faster loadable than that, but I guess
that was simply too optimistic.
Hm, then I wanted to check what the file size would be without
compression. So I exported to gdtb, setting compression to 0 (all in the
GUI); after a while I get an error window: " .... error zipping" !?
thanks
sven
3:34 p.m.
On Mon, 11 Jan 2021, Sven Schreiber wrote:
Am 11.01.2021 um 19:38 schrieb Allin Cottrell:
[...]
>
> Please try the test under "data-io" at
> https://sourceforge.net/p/gretl/workspace/ci/master/tree/
Sorry, observing this on Windows here I just did a much cruder
comparison: Loading the same dataset from gdtb (new I think)
If by "new" you mean the new pure-binary format, a gretl data file
will be in that format only if saved via CLI with the --purebin
option.
and from gdt. BTW gdt in this heavily unbalanced case is smaller
at only 18MB compared to 26MB before. I get:
- 6.6s for gdtb
- 14.7s for gdt
So yes, the binary stuff still helps a lot. It was just my expectation
that 26MB somehow "should" be faster loadable than that, but I guess
that was simply too optimistic.
What are the specs of the machine in question? On my 2014-vintage
desktop the 26MB load time is less then a second for both formats
(and 20 milliseconds for the new binary format).
Hm, then I wanted to check what the file size would be without
compression. So I exported to gdtb, setting compression to 0 (all in the
GUI); after a while I get an error window: " .... error zipping" !?
I can't reproduce that, either on Linux or Windows.
Allin
4:05 p.m.
Am 12.01.2021 um 16:34 schrieb Allin Cottrell:
On Mon, 11 Jan 2021, Sven Schreiber wrote:
> Sorry, observing this on Windows here I just did a much cruder
> comparison: Loading the same dataset from gdtb (new I think)
If by "new" you mean the new pure-binary format, a gretl data file
will be in that format only if saved via CLI with the --purebin option.
Sorry for the confusion: No, I meant that in the GUI on saving I'm
getting a choice how to represent missings, "new" or old/compatible way.
I chose new.
What are the specs of the machine in question? On my 2014-vintage
desktop the 26MB load time is less then a second for both formats
(and 20 milliseconds for the new binary format).
That PC is a veteran Ivybridge (I think) quadcore i5 with 8GB memory and
SSD drives.
I just checked on a newer i7/16GB Laptop and yes it is somewhat faster
(also Win10). But I guess the relevant property might be that gretl's
memory consumption when loading the 26MB gdtb file goes up from <10MB to
700MB. So I suppose it's not so much the disk file size but
probably
the heavily compressed variables with many discrete-valued or dummy
series in the dataset. (This was originally a Stata file.) And/or the
representation of missings.
> Hm, then I wanted to check what the file size would be without
> compression. So I exported to gdtb, setting compression to 0 (all in the
> GUI); after a while I get an error window: " .... error zipping" !?
I can't reproduce that, either on Linux or Windows.
On this laptop now I don't see this anymore, either. Could be that the
system partition on the other PC was almost full. So the non-compressed
gdtb file indeed takes up 745MB.
OK, that was with "new missings representation". If instead I use the
old missings choice, that brings me back to the question about
compression, because:
- old missings, compression level 0, gdtb: 25900KB
- old missings, compression level 1, gdtb: 25900KB
How is this same number of KB possible?
(Sorry, can't send the file, it's sensitive data.)
thanks
sven
12:20 a.m.
On Tue, 12 Jan 2021, Sven Schreiber wrote:
If I use the old missings choice, that brings me back to the
question about compression, because:
- old missings, compression level 0, gdtb: 25900KB
- old missings, compression level 1, gdtb: 25900KB
How is this same number of KB possible?
That'll happen if, when minimal (> 0) computational resources are
devoted to finding matching strings across successive 32k blocks of
data, nothing useful is found. See Mark Adler's response at
https://stackoverflow.com/questions/40595352/compression-ratios-for-diffe...
But you should examine the size of the output at byte resolution
rather than KB to see if there's really any difference.
Allin
2:46 p.m.
Am 13.01.2021 um 01:20 schrieb Allin Cottrell:
On Tue, 12 Jan 2021, Sven Schreiber wrote:
Hm, previously I had written that the uncompressed file (gdtb, new
missing style) takes up 745MB, but now (back on the first machine again)
I'm actually unable to replicate such a large file...
> If I use the old missings choice, that brings me back to the
question
> about compression, because:
>
> - old missings, compression level 0, gdtb: 25900KB
>
> - old missings, compression level 1, gdtb: 25900KB
>
> How is this same number of KB possible?
That'll happen if, when minimal (> 0) computational resources are
devoted to finding matching strings across successive 32k blocks of
data, nothing useful is found. See Mark Adler's response at
OK, thanks. Of course for my concrete example it's all not very
important, it's just about checking whether the user choices are
correctly given and honored.
But I couldn't find any explanation about "old" vs. "new" missings
in
the gdtb format, was it about using gretl's own encoding vs. the IEEE
missing standard?
thanks
sven
4:48 p.m.
On Wed, 13 Jan 2021, Sven Schreiber wrote:
Am 13.01.2021 um 01:20 schrieb Allin Cottrell:
> On Tue, 12 Jan 2021, Sven Schreiber wrote:
Hm, previously I had written that the uncompressed file (gdtb, new
missing style) takes up 745MB, but now (back on the first machine again)
I'm actually unable to replicate such a large file...
>> If I use the old missings choice, that brings me back to the question
>> about compression, because:
>>
>> - old missings, compression level 0, gdtb: 25900KB
>>
>> - old missings, compression level 1, gdtb: 25900KB
>>
>> How is this same number of KB possible?
>
> That'll happen if, when minimal (> 0) computational resources are
> devoted to finding matching strings across successive 32k blocks of
> data, nothing useful is found. See Mark Adler's response at
OK, thanks. Of course for my concrete example it's all not very
important, it's just about checking whether the user choices are
correctly given and honored.
But I couldn't find any explanation about "old" vs. "new"
missings in
the gdtb format, was it about using gretl's own encoding vs. the IEEE
missing standard?
Yes, the "old" treatment of missing values in a gdtb save means it
uses the definition of NA in gretl < 2018c, while the "new" (now
more than 2 years old) version uses NaN. The only reason to use the
old version is if you want the file to be readable by gretl 2018b or
earlier. It may be about time to retire that option?
Allin
4:51 p.m.
Am 13.01.2021 um 17:48 schrieb Allin Cottrell:
On Wed, 13 Jan 2021, Sven Schreiber wrote:
>
> OK, thanks. Of course for my concrete example it's all not very
> important, it's just about checking whether the user choices are
> correctly given and honored.
>
> But I couldn't find any explanation about "old" vs. "new"
missings in
> the gdtb format, was it about using gretl's own encoding vs. the IEEE
> missing standard?
Yes, the "old" treatment of missing values in a gdtb save means it uses
the definition of NA in gretl < 2018c, while the "new" (now more than 2
years old) version uses NaN. The only reason to use the old version is
if you want the file to be readable by gretl 2018b or earlier. It may be
about time to retire that option?
Yes, what about first thing after release?
cheers
sven
5:02 p.m.
Am 13.01.2021 um 01:20 schrieb Allin Cottrell:
On Tue, 12 Jan 2021, Sven Schreiber wrote:
> If I use the old missings choice, that brings me back to the question
> about compression, because:
>
> - old missings, compression level 0, gdtb: 25900KB
>
> - old missings, compression level 1, gdtb: 25900KB
>
> How is this same number of KB possible?
That'll happen if, when minimal (> 0) computational resources are
devoted to finding matching strings across successive 32k blocks of
data, nothing useful is found. See Mark Adler's response at
https://stackoverflow.com/questions/40595352/compression-ratios-for-diffe...
OK, picking this up in the context of gdtb format (but not with
different missing representation). With a Jan 19th snapshot I have saved
a file always in the current gdtb format, and with compression settings
0 (none/min), 5, 9 (max), and I get exactly the same resulting file
size. Am I too naive to expect a difference between settings 0 and 9 ?
Or if the setting doesn't apply to the current gdtb format, perhaps it
should be greyed out? I'm noticing that the reference for the store
command doesn't mention any compression for gdtb (only for gdt).
thanks
sven
5:05 p.m.
On Mon, 11 Jan 2021, Sven Schreiber wrote:
Hi,
another minor glitch: I had a large Stata dta file and imported it into
gretl - this actually worked flawlessly via dragging-and-dropping (on
Windows). Then I went on to save it; in the save dialog gretl inserted
the extension "gdt" automatically, and when I changed the format to the
binary gdtb type in the drop-down box at the bottom, in the end I had a
file ending in .gdt.gdtb.
Obviously very easy to correct manually in various ways along the steps
taken, but maybe now we want to shake out even minor stuff like this.
Yes, let's. That issue is now fixed in git: we defer the
auto-addition of a suffix until we know which format is selected,
gdt or gdtb.
Allin
1424
days inactive
1440
days old
11 comments
2 participants
participants (2)
-
Allin Cottrell -
Sven Schreiber