7:39 p.m.
Dear all,
I am currently working with a 2.8GB csv data file which I stored as a
gdtb file (compressed to about 170MB). However, trying to open the gdtb
file yields an error (see attached screenshot) with the message that the
data is not compressed in a valid manner. Smaller gdtb files work fine
though.
If I remember correctly, I also obtained a message refering to some 4GB
limit this afternoon -- however I cannot replicate this.
The issue occurs on Ubuntu 18.04 using latest git.
Best,
Artur
8:05 p.m.
Sorry, for cross-posting. The message I received this morning I just
attached again.
So it seems this issue is related to zip as a format which has a 4gb
limit. Is there a way to circumvent this - for instance by switching to
gzip or so (just in case the user can control this).
Artur
Am 01.10.19 um 21:39 schrieb Artur Tarassow:
Dear all,
I am currently working with a 2.8GB csv data file which I stored as a
gdtb file (compressed to about 170MB). However, trying to open the
gdtb file yields an error (see attached screenshot) with the message
that the data is not compressed in a valid manner. Smaller gdtb files
work fine though.
If I remember correctly, I also obtained a message refering to some
4GB limit this afternoon -- however I cannot replicate this.
The issue occurs on Ubuntu 18.04 using latest git.
Best,
Artur
8:22 p.m.
Am 01.10.2019 um 22:05 schrieb Artur Tarassow:
Sorry, for cross-posting. The message I received this morning I just
attached again.
So it seems this issue is related to zip as a format which has a 4gb
limit. Is there a way to circumvent this - for instance by switching
to gzip or so (just in case the user can control this).
Hi Artur, I believe the 4GB limit is inherent in the zip format, not a
limitation of the tool(s). The successor format seems to be zip64, maybe
gretl should switch to using that.
In the meantime, maybe the file can be split up?
cheers
sven
8:32 p.m.
Am 01.10.19 um 22:22 schrieb Sven Schreiber:
Am 01.10.2019 um 22:05 schrieb Artur Tarassow:
> Sorry, for cross-posting. The message I received this morning I just
> attached again.
>
> So it seems this issue is related to zip as a format which has a 4gb
> limit. Is there a way to circumvent this - for instance by switching
> to gzip or so (just in case the user can control this).
Hi Artur, I believe the 4GB limit is inherent in the zip format, not a
limitation of the tool(s). The successor format seems to be zip64, maybe
gretl should switch to using that.
Hi Sven,
it seems that gretl uses PKZIP. That's what I found in the manual and
also some hint on Allins homepage:
http://ricardo.ecn.wfu.edu/~cottrell/pkzip.html
I could not find anything on whether newer versions of PKZIP supports a
file type like zip64.
In the meantime, maybe the file can be split up?
Yes, would be a strategy. I could then join the n-1 splited data sets.
Originally I had the following strategy in mind:
1) Reduce the full data set to a single variable which I use as a filter
to select only some cross-sectional units (around 58,000 out of 110,000).
2) Re-join about 10 series to the remaining 58,000 cross-sectional units
(T=639). However, joining a _single_ series (--ikey has 2 identifying
variables) already takes almost 1h -- so this is no solution for me ;-)
RAM is no issue but loading and storing big CSVs takes quite long. So
gdtb is the recommended alternative.
Artur
12:48 a.m.
On Tue, 1 Oct 2019, Artur Tarassow wrote:
I am currently working with a 2.8GB csv data file which I stored
as a gdtb file (compressed to about 170MB). However, trying to
open the gdtb file yields an error (see attached screenshot) with
the message that the data is not compressed in a valid manner.
Smaller gdtb files work fine though.
There are two code paths for zipping/unzipping in gretl (other than
just straight gzip compression or extraction of a single file, which
is handled by the zlib API).
1) The gretlzip plugin module, which is based on the code for the
Info-ZIP executable "zip" (Mark Adler et al), version 2.31. This is
now somewhat dated.
2) Libgsf-1 (gsf = gnome structured file, used as back-end for
gnumeric).
I suspect that 1) may be subject to a 4GB limit but 2) is not.
In years past we considered use of libgsf as experimental, and you
had to specifically request it in a gretl build by passing
--with-gsf to the configure script. I guess it's time to change
that. Now (current git) if libgsf >= 1.14.31 is found when
configuring, it will be used by preference. (Of course, you'd need
the "dev" or "devel" libgsf package.)
Allin
2:06 p.m.
Am 02.10.19 um 02:48 schrieb Allin Cottrell:
On Tue, 1 Oct 2019, Artur Tarassow wrote:
> I am currently working with a 2.8GB csv data file which I stored as a
> gdtb file (compressed to about 170MB). However, trying to open the
> gdtb file yields an error (see attached screenshot) with the message
> that the data is not compressed in a valid manner. Smaller gdtb files
> work fine though.
There are two code paths for zipping/unzipping in gretl (other than just
straight gzip compression or extraction of a single file, which is
handled by the zlib API).
1) The gretlzip plugin module, which is based on the code for the
Info-ZIP executable "zip" (Mark Adler et al), version 2.31. This is now
somewhat dated.
2) Libgsf-1 (gsf = gnome structured file, used as back-end for gnumeric).
I suspect that 1) may be subject to a 4GB limit but 2) is not.
In years past we considered use of libgsf as experimental, and you had
to specifically request it in a gretl build by passing --with-gsf to the
configure script. I guess it's time to change that. Now (current git) if
libgsf >= 1.14.31 is found when configuring, it will be used by
preference. (Of course, you'd need the "dev" or "devel" libgsf
package.)
Thanks for the information, Allin.
Using libgsf works fine here on Ubuntu 19.04 using large files.. I use
the --gzipped option now which reduces file size massively.
Best,
Artur
2:33 p.m.
On Wed, 2 Oct 2019, Artur Tarassow wrote:
Am 02.10.19 um 02:48 schrieb Allin Cottrell:
> On Tue, 1 Oct 2019, Artur Tarassow wrote:
>
> > I am currently working with a 2.8GB csv data file which I stored as a gdtb
> > file (compressed to about 170MB). However, trying to open the gdtb file
> > yields an error (see attached screenshot) with the message that the data
> > is not compressed in a valid manner. Smaller gdtb files work fine though.
>
> There are two code paths for zipping/unzipping in gretl (other than just
> straight gzip compression or extraction of a single file, which is handled
> by the zlib API).
>
> 1) The gretlzip plugin module, which is based on the code for the Info-ZIP
> executable "zip" (Mark Adler et al), version 2.31. This is now somewhat
> dated.
>
> 2) Libgsf-1 (gsf = gnome structured file, used as back-end for gnumeric).
>
> I suspect that 1) may be subject to a 4GB limit but 2) is not.
>
> In years past we considered use of libgsf as experimental, and you had to
> specifically request it in a gretl build by passing --with-gsf to the
> configure script. I guess it's time to change that. Now (current git) if
> libgsf >= 1.14.31 is found when configuring, it will be used by preference.
> (Of course, you'd need the "dev" or "devel" libgsf
package.)
Thanks for the information, Allin.
I'm currently working on replacing the Info-ZIP code with libgsf for
our gretl builds for Windows and Mac; that should be in the snapshots
before long.
Allin
1921
days inactive
1922
days old
6 comments
3 participants
participants (3)
-
Allin Cottrell -
Artur Tarassow -
Sven Schreiber