12:30 a.m.
Hello everybody,
I have a data-set with some non-numeric values according to gretl (but not to CSVed).
[1]: Problematic values (examples):
4.13108e-312 5.27295e-270 4.3172e-227 1.51922e-184 3.97655e-142 1.42149e-144 1.94409e-143 2.79222e-144
(These values stem from uninitialised double variables in my c++ model, I guess).
Is there an option unbeknown to me to simply substitute such non-numerics by Gretl
"NA" directly with the import? I.e. make the error messages quiet and gretl
process nonetheless?
I cannot find one and an old request (http://sourceforge.net/p/gretl/feature-requests/47/)
is long closed...
----
Background:
I have a relative large data-set (57,970 rows x ~1,000 columns) from a simulation
experiment (panel data), distributed over several paths and files. I have written a python
script that produces a nice enough import script for gretl (listing all the paths, the
first one via open the next ones via append) which works fine... except when some
not-a-number values are present, like "1.#QNAN" or [1], which occurs sometimes.
I would prefer not to change the data (or a copy of the data) because a) this probably
takes a lot of time (the total size is about 800MB and more to come) and b) Gretl does a
good job in not mixing things up...
----
Thanks for your time!
Frederik
------------
Frederik Schaff, Dipl.-Volkswirt
University of Hagen
Department of Business and Economics
Chair of Economic Theory (Prof. A. Endres)
Universitätsstraße 11 (TGZ)
58097 Hagen
Phone: +49 (0) 2331 987-4454
E-Mail: Frederik.Schaff(a)FernUni-Hagen.de
http://www.fernuni-hagen.de/wirtschaftstheorie/en/team/frederik.schaff.shtml
8:30 a.m.
On Sun, 27 Sep 2015, Schaff, Frederik wrote:
Hello everybody,
I have a data-set with some non-numeric values according to gretl (but not to CSVed).
[1]: Problematic values (examples):
4.13108e-312 5.27295e-270 4.3172e-227 1.51922e-184 3.97655e-142 1.42149e-144 1.94409e-143 2.79222e-144
(These values stem from uninitialised double variables in my c++ model, I guess).
Is there an option unbeknown to me to simply substitute such
non-numerics by Gretl "NA" directly with the import? I.e. make the error
messages quiet and gretl process nonetheless?
I cannot find one and an old request
(http://sourceforge.net/p/gretl/feature-requests/47/) is long closed...
In principle, anything is possible, since in Hansl you get several
string-related function for importing your file as a big string and
manipulating it later.
However, if you're using a Unix-like system (Linux, BSD, OSX...) you might
want to pre-process the data and produce a more user-friendly csv file via
the trusty, old, all-powerful unix tools. Or at least, that's what *I*
would do. Something like
sed yourdata.csv -e 's/[0-9].[0-9]*[Ee]-[1-3][0-9]2/NA/g' > cleandata.csv
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
1:05 p.m.
Thank you for your time!
I am using a windows machine, unfortunately (and not used to using linux/unix os)... I
took a look again into gretl and hansl guides, but with join and/or append I could not
find such an option. However your answer already suggested that there is no
"easy" way I missed. So I'll set out to pre-process the data via python or
so.
Perhaps, if it is not much of a problem, such a "non-numeric-->NA" import
option would be something for a future version?
Regards and thanks for the quick answer
Frederik
------------
Frederik Schaff, Dipl.-Volkswirt
University of Hagen
Department of Business and Economics
Chair of Economic Theory (Prof. A. Endres)
Universitätsstraße 11 (TGZ)
58097 Hagen
Phone: +49 (0) 2331 987-4454
E-Mail: Frederik.Schaff(a)FernUni-Hagen.de
http://www.fernuni-hagen.de/wirtschaftstheorie/en/team/frederik.schaff.shtml
-----Ursprüngliche Nachricht-----
Von: gretl-users-bounces(a)lists.wfu.edu [mailto:gretl-users-bounces@lists.wfu.edu] Im
Auftrag von Riccardo (Jack) Lucchetti
Gesendet: Sonntag, 27. September 2015 10:30
An: Gretl list
Betreff: Re: [Gretl-users] Data Import - non-numeric values
On Sun, 27 Sep 2015, Schaff, Frederik wrote:
Hello everybody,
I have a data-set with some non-numeric values according to gretl (but not to CSVed).
[1]: Problematic values (examples):
4.13108e-312 5.27295e-270 4.3172e-227 1.51922e-184 3.97655e-142 1.42149e-144 1.94409e-143 2.79222e-144
(These values stem from uninitialised double variables in my c++ model, I guess).
Is there an option unbeknown to me to simply substitute such
non-numerics by Gretl "NA" directly with the import? I.e. make the
error messages quiet and gretl process nonetheless?
I cannot find one and an old request
(http://sourceforge.net/p/gretl/feature-requests/47/) is long closed...
In principle, anything is possible, since in Hansl you get several string-related function
for importing your file as a big string and manipulating it later.
However, if you're using a Unix-like system (Linux, BSD, OSX...) you might want to
pre-process the data and produce a more user-friendly csv file via the trusty, old,
all-powerful unix tools. Or at least, that's what *I* would do. Something like
sed yourdata.csv -e 's/[0-9].[0-9]*[Ee]-[1-3][0-9]2/NA/g' > cleandata.csv
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
4:21 p.m.
Am 27.09.2015 um 15:05 schrieb Schaff, Frederik:
Thank you for your time!
I am using a windows machine, unfortunately (and not used to using
linux/unix os)... I took a look again into gretl and hansl guides,
but with join and/or append I could not find such an option. H
This may be a stupid question, but why exactly is a value like
4.13108e-312 problematic for gretl? Or to rephrase the question, can you
be more specific on what actually doesn't work and what the error
message is?
'join' is quite powerful, so I would be optimistic that something can be
done with your data. But so far I haven't really understood the problem.
Maybe you can provide a snippet of 5 or 10 lines of what your csv file
really looks like and what you want to have as a result in gretl.
cheers,
sven
4:32 p.m.
On Sun, 27 Sep 2015, Schaff, Frederik wrote:
Hello everybody,
I have a data-set with some non-numeric values according to gretl
(but not to CSVed). [1]: Problematic values (examples):
4.13108e-312 5.27295e-270 4.3172e-227 1.51922e-184 3.97655e-142
1.42149e-144 1.94409e-143 2.79222e-144 (These values stem from
uninitialised double variables in my c++ model, I guess).
Interesting. Some of these numbers are in the "underflow" range where
their handling by the C-library function strtod() -- which gretl uses
as its test for numeric vs non-numeric -- is "implementation defined",
meaning that standard-conformant C libraries may legitimately behave
differently.
With glibc 2.22 (Arch Linux) the first string above, "4.13108e-312",
is converted to a non-zero value OK but errno is set to E_RANGE
("Numerical result out of range") because the result is "subnormal".
Gretl notices the non-zero errno and takes the value to be
non-numeric. (All the other values above are converted without errno
being set.)
We could make our test a little more "forgiving" by flagging an
underflowing value as non-numeric only if errno = E_RANGE _and_ the
value returned by strtod is zero.
Is there an option unbeknown to me to simply substitute such
non-numerics by Gretl "NA" directly with the import? I.e. make the
error messages quiet and gretl process nonetheless?
No, and I don't really think that's a good idea. The presence of
non-numeric values in a mostly numeric column generally indicates that
a data file is broken (whether truly non-numeric values have somehow
crept in or, as in this case, uninitialized floating-point values have
been spewed into the file.) And in that case it's up to the user to
fix the data.
Allin Cottrell
4:57 p.m.
Ok. Here is the "script output" and also a part of one file that is problematic.
The "observation" in the script-out refers to the "config" variable
(first column)-1 (because configs range from 0 to ...), there are three problematic values
in the file.
Thanks
Frederik
-----Ursprüngliche Nachricht-----
Von: gretl-users-bounces(a)lists.wfu.edu [mailto:gretl-users-bounces@lists.wfu.edu] Im
Auftrag von Allin Cottrell
Gesendet: Sonntag, 27. September 2015 18:33
An: Gretl list
Betreff: Re: [Gretl-users] Data Import - non-numeric values
On Sun, 27 Sep 2015, Schaff, Frederik wrote:
Hello everybody,
I have a data-set with some non-numeric values according to gretl (but
not to CSVed). [1]: Problematic values (examples):
4.13108e-312 5.27295e-270 4.3172e-227 1.51922e-184 3.97655e-142
1.42149e-144 1.94409e-143 2.79222e-144 (These values stem from
uninitialised double variables in my c++ model, I guess).
Interesting. Some of these numbers are in the "underflow" range where their
handling by the C-library function strtod() -- which gretl uses as its test for numeric vs
non-numeric -- is "implementation defined", meaning that standard-conformant C
libraries may legitimately behave differently.
With glibc 2.22 (Arch Linux) the first string above, "4.13108e-312", is
converted to a non-zero value OK but errno is set to E_RANGE ("Numerical result out
of range") because the result is "subnormal".
Gretl notices the non-zero errno and takes the value to be non-numeric. (All the other
values above are converted without errno being set.)
We could make our test a little more "forgiving" by flagging an underflowing
value as non-numeric only if errno = E_RANGE _and_ the value returned by strtod is zero.
Is there an option unbeknown to me to simply substitute such
non-numerics by Gretl "NA" directly with the import? I.e. make the
error messages quiet and gretl process nonetheless?
No, and I don't really think that's a good idea. The presence of non-numeric
values in a mostly numeric column generally indicates that a data file is broken (whether
truly non-numeric values have somehow crept in or, as in this case, uninitialized
floating-point values have been spewed into the file.) And in that case it's up to the
user to fix the data.
Allin Cottrell
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
5:21 p.m.
On Sun, 27 Sep 2015, Schaff, Frederik wrote:
Ok. Here is the "script output" and also a part of one file
that is
problematic. The "observation" in the script-out refers to the
"config" variable (first column)-1 (because configs range from 0 to
...), there are three problematic values in the file.
Thanks. The correct fix here it to edit the data file to turn the
uninitialized values into NAs (or perhaps 0s, depending on how they
came about?). Or better, to fix the process that produced the data
file and re-run it.
Note that whether such values are taken as "numeric" or not will in
general depend on the C library in use. But either way they're wrong
and have to be changed. _If_ you can get such values into gretl as
numeric, you could fix them via something like:
foo = (abs(foo) > 0 && abs(foo) < 1.0e-100)? NA : foo
where "foo" is the name of the series to be fixed and we're assuming
that non-zero observations with absolute value less than 10^{-100} are
garbage. This is not very reliable, however, as it's _possible_ that
some uninitialized doubles happen to fall in the "normal" range and so
escape correction. The only truly reliable procedure is to fix the
process that generated the data in the first place.
Allin Cottrell
7:02 p.m.
On Sun, 27 Sep 2015, Allin Cottrell wrote:
Note that whether [uninitialized] values are taken as
"numeric" or
not will in general depend on the C library in use. But either way
they're wrong and have to be changed. _If_ you can get such values
into gretl as numeric, you could fix them via something like:
foo = (abs(foo) > 0 && abs(foo) < 1.0e-100)? NA : foo
where "foo" is the name of the series to be fixed and we're
assuming that non-zero observations with absolute value less than
10^{-100} are garbage. This is not very reliable, however, as it's
_possible_ that some uninitialized doubles happen to fall in the
"normal" range and so escape correction.
After a little testing, let me rephrase that: it's more than
"possible", it's highly probable.
I wrote a little test C program which created an array of 2048
"doubles", uninitialized. For each such value I printed it into a
string variable using sprintf() with the "%g" conversion then tried
reading it back into a double using strtod(). I counted the cases
where strtod() raised the ERANGE error: 271 out of 2048. So in this
case, at least, the great majority of garbage values appeared to be
"fine": properly numeric and not subnormal.
So here's a big WARNING: on no account should one let uninitialized
values get printed into a file for use in econometric analysis.
There's no half-way reliable method for clearing them out.
[Just as a footnote: a "subnormal" number (also known as
"denormalized") is one that's too close to zero to be represented as
a C "double" to anything like the usual precision. And there's
absolutely no guarantee that the random bits in an uninitialized
double will correspond to a subnormal number.]
Allin Cottrell
5:32 p.m.
Hi there,
thanks very much Allen! I'll take the advice to heart. Fortunately in the case where
these garbage values are "created" a part of the analysis (corresponding to
these values) has not been conducted and that is flagged (in another
"non-garbage" variable), so I can post-process these values. What are the
"maximal" values gretl takes as import? +-1e100 and +-1e-100?
Regards
Frederik
-----Ursprüngliche Nachricht-----
Von: gretl-users-bounces(a)lists.wfu.edu [mailto:gretl-users-bounces@lists.wfu.edu] Im
Auftrag von Allin Cottrell
Gesendet: Sonntag, 27. September 2015 21:02
An: Gretl list
Betreff: Re: [Gretl-users] Data Import - non-numeric values
On Sun, 27 Sep 2015, Allin Cottrell wrote:
Note that whether [uninitialized] values are taken as
"numeric" or not
will in general depend on the C library in use. But either way they're
wrong and have to be changed. _If_ you can get such values into gretl
as numeric, you could fix them via something like:
foo = (abs(foo) > 0 && abs(foo) < 1.0e-100)? NA : foo
where "foo" is the name of the series to be fixed and we're assuming
that non-zero observations with absolute value less than 10^{-100} are
garbage. This is not very reliable, however, as it's _possible_ that
some uninitialized doubles happen to fall in the "normal" range and so
escape correction.
After a little testing, let me rephrase that: it's more than "possible",
it's highly probable.
I wrote a little test C program which created an array of 2048 "doubles",
uninitialized. For each such value I printed it into a string variable using sprintf()
with the "%g" conversion then tried reading it back into a double using
strtod(). I counted the cases where strtod() raised the ERANGE error: 271 out of 2048. So
in this case, at least, the great majority of garbage values appeared to be
"fine": properly numeric and not subnormal.
So here's a big WARNING: on no account should one let uninitialized values get printed
into a file for use in econometric analysis.
There's no half-way reliable method for clearing them out.
[Just as a footnote: a "subnormal" number (also known as
"denormalized") is one that's too close to zero to be represented as a C
"double" to anything like the usual precision. And there's absolutely no
guarantee that the random bits in an uninitialized double will correspond to a subnormal
number.]
Allin Cottrell
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
8:08 p.m.
On Mon, 28 Sep 2015, Schaff, Frederik wrote:
Hi there,
thanks very much Allen! I'll take the advice to heart. Fortunately in
the case where these garbage values are "created" a part of the analysis
(corresponding to these values) has not been conducted and that is
flagged (in another "non-garbage" variable), so I can post-process these
values. What are the "maximal" values gretl takes as import? +-1e100 and
+-1e-100?
If I were you, I'd use a conventional vaule for "missing" (say,
-99999.99999), which would be subsequently easy to convert to a "proper"
missing entry via the gretl "setmiss" command.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
8:37 p.m.
On Mon, 28 Sep 2015, Schaff, Frederik wrote:
thanks very much Allen! I'll take the advice to heart.
Fortunately
in the case where these garbage values are "created" a part of the
analysis (corresponding to these values) has not been conducted
and that is flagged (in another "non-garbage" variable), so I can
post-process these values. What are the "maximal" values gretl
takes as import? +-1e100 and +-1e-100?
Gretl accepts the judgment of the C library on numerical underflow
or overflow. On the big side we can be fairly definite: anything
less than 1.79769e308 should be fine. On the close-to-zero side
numbers greater in absolute value than 1e-308 should be OK for most
C libraries.
Allin
6:43 p.m.
Hello,
I checked my code a bit more. Apparently it was not the case that some un-initialised
garbage was saved but that floating-point errors must have become evident. The original
series of which I calculated the order stats (lq = lower quartile -- lq_lq was the lower
quartile of the series of the lower quartile of the stat "Hunger") was an ARMA
like process, adding and multiplying "close to zeros" (see attached file)...
Just one thing where I did not control for such behaviour. :/
See e.g. Izquierdo, Luis R.; Polhill, J. Gary (2006): Is Your Model Susceptible to
Floating-Point Errors? In: JASS 9 (4), S. 4. http://jasss.soc.surrey.ac.uk/9/4/4.html
Regards
Frederik
-----Ursprüngliche Nachricht-----
Von: gretl-users-bounces(a)lists.wfu.edu [mailto:gretl-users-bounces@lists.wfu.edu] Im
Auftrag von Allin Cottrell
Gesendet: Montag, 28. September 2015 22:38
An: Gretl list
Betreff: Re: [Gretl-users] Data Import - non-numeric values
On Mon, 28 Sep 2015, Schaff, Frederik wrote:
thanks very much Allen! I'll take the advice to heart.
Fortunately in
the case where these garbage values are "created" a part of the
analysis (corresponding to these values) has not been conducted and
that is flagged (in another "non-garbage" variable), so I can
post-process these values. What are the "maximal" values gretl takes
as import? +-1e100 and +-1e-100?
Gretl accepts the judgment of the C library on numerical underflow or overflow. On the big
side we can be fairly definite: anything less than 1.79769e308 should be fine. On the
close-to-zero side numbers greater in absolute value than 1e-308 should be OK for most C
libraries.
Allin
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
3151
days inactive
3153
days old
11 comments
4 participants
participants (4)
-
Allin Cottrell -
Riccardo (Jack) Lucchetti -
Schaff, Frederik -
Sven Schreiber