[Gretl-devel] redefining "NA"
I posted about a possible redefinition of the internal value
corresponding to "NA" or missing in
http://lists.wfu.edu/pipermail/gretl-devel/2018-July/008926.html
(Reminder: there's a short PDF attachment which explains the
associated issues.)
I've now checked the new scheme (NA = NaN internally, rather than NA =
DBL_MAX) on all of my test scripts, all current function packages and
all the official "addons", and everything seems to be OK. But there's
one point that's maybe worth discussing before I "flip the switch" in
git and snapshots.
There's a potential compatibility issue with the gdtb binary data file
format. Backward compatibility will be preserved: there's code in
place to ensure that "new gretl" will handle old gdtb files OK
(converting missing values from DBL_MAX to NaN on reading). But
"forward compatibility" is likely to be at least partially broken.
That is, gdtb files written by "new gretl" (and containing NAs) will
not be handled correctly by "old gretl". Two comments on this:
1. It's easy enough to convert a dataset read by old gretl from a new
gdtb file to old-style NAs. The following script will do the job:
<hansl>
function void na_conv (series *y)
loop i=1..$nobs -q
y[i] = y[i]
endloop
end function
open new.gdtb
list L = dataset
loop foreach i L -q
na_conv(&$i)
endloop
</hansl>
This may look funny but it works fine, based on the way that old gretl
automatically converts non-finite values to NA on assignment to
series. A simpler variant of the na_conv function would also work:
function void na_conv (series *y)
y = y
end function
but this version would lose the descriptive labels on the series.
2. We could preserve forward as well as backward compatibility if we
were to convert new-style NAs to DBL_MAX on writing gdtb files. But
one of the benefits of the gdtb format is very fast input-output for
big datasets, and that would be compromised if we had to perform
NA conversion on all writes and reads.
My feeling is that it's worth paying the incompatibility price,
particularly since there's a workaround available.
Allin