2:18 p.m.
Following discussions between Jack and myself, here's a sort of
"RFC" for changing the internal definition of "NA" in gretl, which we
believe can be done with full backward compatibility.
Little PDF attached.
Allin
2:38 p.m.
Am 24.07.2018 um 16:18 schrieb Allin Cottrell:
Following discussions between Jack and myself, here's a sort of
"RFC" for changing the internal definition of "NA" in gretl, which we
believe can be done with full backward compatibility.
Little PDF attached.
Thanks for the excellent exposition of the problem. One thing I don't
understand from the document, however, is what would happen in the end
with the NA * 0 = 0 feature when NA becomes nan. AFAICS your point is
that it would be a nice feature but we don't have it currently, anyway,
so let's forget about it forever? (Not saying I'd be against this, just
trying to understand for now.)
thanks,
sven
3:27 p.m.
On Tue, 24 Jul 2018, Sven Schreiber wrote:
Am 24.07.2018 um 16:18 schrieb Allin Cottrell:
> Following discussions between Jack and myself, here's a sort of
> "RFC" for changing the internal definition of "NA" in gretl,
which we
> believe can be done with full backward compatibility.
>
> Little PDF attached.
Thanks for the excellent exposition of the problem. One thing I don't
understand from the document, however, is what would happen in the end with
the NA * 0 = 0 feature when NA becomes nan. AFAICS your point is that it
would be a nice feature but we don't have it currently, anyway, so let's
forget about it forever? (Not saying I'd be against this, just trying to
understand for now.)
Yes, I think "nice feature but we don't have it currently" is about
right. Consider the following (abdata has lots of missing values,
including the very first observation):
<hansl>
open abdata.gdt
WAGE[1] = WAGE[1] * 0
smpl 1:1 1:9
print WAGE --byobs
series bad = log(-WAGE)
bad[1] = bad[1] * 0
print bad --byobs
</hansl>
The first "print" shows that the leading NA for WAGE has indeed been
changed to 0 (defensibly). But the second "print" reveals that bad[1]
also gets changed to 0, which is not so defensible. That happens
because the nans in "bad" are automatically mapped to NA as things
stand.
Under the proposed scheme NAs (= nans) would not turn into zeros on
multiplication by zero -- but the misszero() function would still be
available to do that job.
Allin
5:30 p.m.
Am 24.07.2018 um 17:27 schrieb Allin Cottrell:
> Am 24.07.2018 um 16:18 schrieb Allin Cottrell:
>> Following discussions between Jack and myself, here's a sort of
>> "RFC" for changing the internal definition of "NA" in gretl,
which
>> we believe can be done with full backward compatibility.
>>
Yes, I think "nice feature but we don't have it currently" is about
right. Consider the following (abdata has lots of missing values,
including the very first observation):
<hansl>
open abdata.gdt
WAGE[1] = WAGE[1] * 0
...
The first "print" shows that the leading NA for WAGE has
indeed been
changed to 0 (defensibly).
...
Under the proposed scheme NAs (= nans) would not turn into zeros on
multiplication by zero -- but the misszero() function would still be
available to do that job.
OK, but doesn't this consitute a counterexample to
the claim of full
backward compatibility? This doesn't have to be bad, given the perhaps
kind of flaky behavior currently, but it would change something, no?
thanks,
sven
5:43 p.m.
On Tue, 24 Jul 2018, Sven Schreiber wrote:
Am 24.07.2018 um 17:27 schrieb Allin Cottrell:
>
>> Am 24.07.2018 um 16:18 schrieb Allin Cottrell:
>>> Following discussions between Jack and myself, here's a sort of
>>> "RFC" for changing the internal definition of "NA" in
gretl, which we
>>> believe can be done with full backward compatibility.
>>>
> Yes, I think "nice feature but we don't have it currently" is about
right.
> Consider the following (abdata has lots of missing values, including the
> very first observation):
>
> <hansl>
> open abdata.gdt
> WAGE[1] = WAGE[1] * 0
>
...
> The first "print" shows that the leading NA for WAGE has indeed been
> changed to 0 (defensibly).
...
>
> Under the proposed scheme NAs (= nans) would not turn into zeros on
> multiplication by zero -- but the misszero() function would still be
> available to do that job.
OK, but doesn't this consitute a counterexample to the claim of full
backward compatibility?
Yes, you're right. I hadn't thought that through all the way.
This doesn't have to be bad, given the perhaps kind of flaky
behavior currently, but it would change something, no?
It raises a compatibility issue if there are scripts out there that
depend on multiplication by 0 to neutralize NAs. I guess that's
unlikely, given misszero(), but clearly we'd have to check all
function packages, at least.
Allin
11:28 p.m.
On Tue, 24 Jul 2018, Allin Cottrell wrote:
On Tue, 24 Jul 2018, Sven Schreiber wrote:
> Am 24.07.2018 um 17:27 schrieb Allin Cottrell:
>>
>> Under the proposed scheme NAs (= nans) would not turn into zeros on
>> multiplication by zero -- but the misszero() function would still be
>> available to do that job.
>
> OK, but doesn't this consitute a counterexample to the claim of full
> backward compatibility?
Yes, you're right. I hadn't thought that through all the way.
One more thought on this point. We _could_, if we wanted to,
preserve the current "NA * 0 = 0" rule for arithmetic on scalars and
series in its semi-broken form (meaning, it applies without regard
to whether the NAs are "real missing values" or converted nans or
infinities).
That would not be computationally costly, and it would be maximally
backward-compatible, but I think my preference is to break
compatibility in that one respect, because what we're doing now is
not really defensible. One point I'd forgotten is that comparable
software doesn't support "NA * 0 = 0". Not that we want to slavishly
follow R, but NA * 0 = NA there.
Allin
12:43 p.m.
Am 25.07.2018 um 01:28 schrieb Allin Cottrell:
On Tue, 24 Jul 2018, Allin Cottrell wrote:
> On Tue, 24 Jul 2018, Sven Schreiber wrote:
>
>>
>> OK, but doesn't this consitute a counterexample to the claim of full
>> backward compatibility?
>
That would not be computationally costly, and it would be maximally
backward-compatible, but I think my preference is to break
compatibility in that one respect, because what we're doing now is not
really defensible.
Yes, I tend to agree. (Remembering earlier discussions about the NA/nan
situation, I'd say better late than never, hehe.)
I'm wondering whether I should extract from the function packages a
mailing list of package authors for such occasions, to alert them about
architecture changes like this.
One point I'd forgotten is that comparable software doesn't
support
"NA * 0 = 0". Not that we want to slavishly follow R, but NA * 0 = NA
there.
I think the main point is that this NA * 0 isn't really marketed or
mentioned in the gretl documentation, is it?
Apart from that, what's the status of stuff like table 16.1 in the
guide? It mentions "NA" for matrices instead of NaN, has this been wrong
all along?
thanks,
sven
7:35 p.m.
On Thu, 26 Jul 2018, Sven Schreiber wrote:
Am 25.07.2018 um 01:28 schrieb Allin Cottrell:
> One point I'd forgotten is that comparable software doesn't
> support "NA * 0 = 0". Not that we want to slavishly follow R, but
> NA * 0 = NA there.
(In Stata too, I now see.)
I think the main point is that this NA * 0 isn't really marketed
or mentioned in the gretl documentation, is it?
It is mentioned in section 4.6 of the User's Guide, but no, it's not
exactly "marketed".
Apart from that, what's the status of stuff like table 16.1 in
the
guide? It mentions "NA" for matrices instead of NaN, has this been
wrong all along?
What's shown in Table 16.1 is correct: the NAs therein are the
scalar return values from functions that take a matrix as argument
(e.g. determinant), in case the matrix is empty.
Allin
1:18 p.m.
Am 31.07.2018 um 21:35 schrieb Allin Cottrell:
On Thu, 26 Jul 2018, Sven Schreiber wrote:
> Apart from that, what's the status of stuff like table 16.1 in the
> guide? It mentions "NA" for matrices instead of NaN, has this been
> wrong all along?
What's shown in Table 16.1 is correct: the NAs therein are the scalar
return values from functions that take a matrix as argument (e.g.
determinant), in case the matrix is empty.
OK, right, should have noticed that myself.
I'm noting that two paragraphs further down there appears this: " (All
other matrix operations gen-
erate an error when an empty matrix is given as an argument.)" which
basically contradicts the sentence which follows directly, enumerating
more valid functions.
BTW, in this enumeration which is following the 'diagcat' function is
missing it seems. I can edit the guide eventually (in a week or so), not
right now.
Furthermore I wonder why the following functions couldn't also naturally
return empty matrices when given an empty matrix as argument (but this
is only for sake of consistency, not for a particular need):
- lower
- upper
- sort
- dsort
- mreverse
...possible also -but more complicated- those:
- mshape
- msortby
- trimr
cheers,
sven
3:24 p.m.
On Fri, 3 Aug 2018, Sven Schreiber wrote:
Furthermore I wonder why the following functions couldn't also
naturally
return empty matrices when given an empty matrix as argument (but this is
only for sake of consistency, not for a particular need):
I committed to git a simple modification, so that the following now runs
ok:
<hansl>
e = {}
loop foreach i ginv diag transp vec vech unvech cholesky upper lower sort\
dsort values mreverse
x = $i(e)
print x
endloop
</hansl>
Allin, please review and feel free to scrap it if you don't like it.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
4:16 p.m.
On Fri, 3 Aug 2018, Riccardo (Jack) Lucchetti wrote:
On Fri, 3 Aug 2018, Sven Schreiber wrote:
> Furthermore I wonder why the following functions couldn't also naturally
> return empty matrices when given an empty matrix as argument (but this is
> only for sake of consistency, not for a particular need):
I committed to git a simple modification, so that the following now runs ok:
<hansl>
e = {}
loop foreach i ginv diag transp vec vech unvech cholesky upper lower sort\
dsort values mreverse
x = $i(e)
print x
endloop
</hansl>
Allin, please review and feel free to scrap it if you don't like it.
Sorry to be a spoilsport but I'd kinda rather not make this sort of
change right now. Can we review it carefully after the (hopefully
imminent!) release?
Allin
4:20 p.m.
On Fri, 3 Aug 2018, Allin Cottrell wrote:
On Fri, 3 Aug 2018, Riccardo (Jack) Lucchetti wrote:
> On Fri, 3 Aug 2018, Sven Schreiber wrote:
>
>> Furthermore I wonder why the following functions couldn't also naturally
>> return empty matrices when given an empty matrix as argument (but this is
>> only for sake of consistency, not for a particular need):
>
> I committed to git a simple modification, so that the following now runs
> ok:
>
> <hansl>
> e = {}
> loop foreach i ginv diag transp vec vech unvech cholesky upper lower sort\
> dsort values mreverse
> x = $i(e)
> print x
> endloop
> </hansl>
>
> Allin, please review and feel free to scrap it if you don't like it.
Sorry to be a spoilsport but I'd kinda rather not make this sort of change
right now. Can we review it carefully after the (hopefully imminent!)
release?
Fine by me!
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
4:44 p.m.
Am 03.08.2018 um 18:20 schrieb Riccardo (Jack) Lucchetti:
On Fri, 3 Aug 2018, Allin Cottrell wrote:
> On Fri, 3 Aug 2018, Riccardo (Jack) Lucchetti wrote:
>
>
> Sorry to be a spoilsport but I'd kinda rather not make this sort of
> change right now. Can we review it carefully after the (hopefully
> imminent!) release?
Fine by me!
I'm also all for careful code reviews. Jack, in the meantime you could
spend your energies on the documentation instead, hehe.
cheers,
sven
4:51 p.m.
On Fri, 3 Aug 2018, Sven Schreiber wrote:
Am 03.08.2018 um 18:20 schrieb Riccardo (Jack) Lucchetti:
> On Fri, 3 Aug 2018, Allin Cottrell wrote:
>
>> On Fri, 3 Aug 2018, Riccardo (Jack) Lucchetti wrote:
>>
>>
>> Sorry to be a spoilsport but I'd kinda rather not make this sort of change
>> right now. Can we review it carefully after the (hopefully imminent!)
>> release?
>
> Fine by me!
I'm also all for careful code reviews. Jack, in the meantime you could spend
your energies on the documentation instead, hehe.
Boring, boring!
No, seriously: what do you think is most lacking as of now?
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
4:56 p.m.
Am 03.08.2018 um 18:51 schrieb Riccardo (Jack) Lucchetti:
On Fri, 3 Aug 2018, Sven Schreiber wrote:
> I'm also all for careful code reviews. Jack, in the meantime you
> could spend your energies on the documentation instead, hehe.
Boring, boring!
No, seriously: what do you think is most lacking as of now?
Right now I just meant the little contradiction I pointed out above in
this thread.
But if you seriously ask in more general terms:
"32.5 Forecasts based on VAR models
To be written."
Remember the confusion at the Athens conference where people actually
thought it wouldn't be possible to do forecasts from VARs easily?
thanks,
sven
5:11 p.m.
On Fri, 3 Aug 2018, Riccardo (Jack) Lucchetti wrote:
On Fri, 3 Aug 2018, Allin Cottrell wrote:
> On Fri, 3 Aug 2018, Riccardo (Jack) Lucchetti wrote:
>
>> On Fri, 3 Aug 2018, Sven Schreiber wrote:
>>
>>> Furthermore I wonder why the following functions couldn't also naturally
>>> return empty matrices when given an empty matrix as argument (but this is
>>> only for sake of consistency, not for a particular need):
>>
>> I committed to git a simple modification, so that the following now runs
>> ok:
>>
>> <hansl>
>> e = {}
>> loop foreach i ginv diag transp vec vech unvech cholesky upper lower sort\
>> dsort values mreverse
>> x = $i(e)
>> print x
>> endloop
>> </hansl>
>>
>> Allin, please review and feel free to scrap it if you don't like it.
>
> Sorry to be a spoilsport but I'd kinda rather not make this sort of change
> right now. Can we review it carefully after the (hopefully imminent!)
> release?
Fine by me!
Well, actually... Your changes looked a bit daunting since the diff
was quite large, but I think I've found a way to allow those functions
to work on empty matrices with only minimal change to geneval.c. And
I'm now confident there shouldn't be unwanted side-effects. (There
probably wouldn't have been unwanted side-effects anyway, but that's
clearer now.)
To inspect: compare [7a19ca] with [1b7a11] under History at
https://sourceforge.net/p/gretl/git/ci/master/tree/lib/src/geneval.c
Allin
10:08 a.m.
On Fri, 3 Aug 2018, Sven Schreiber wrote:
Am 03.08.2018 um 18:51 schrieb Riccardo (Jack) Lucchetti:
> On Fri, 3 Aug 2018, Sven Schreiber wrote:
>
>> I'm also all for careful code reviews. Jack, in the meantime you could
>> spend your energies on the documentation instead, hehe.
>
> Boring, boring!
>
> No, seriously: what do you think is most lacking as of now?
Right now I just meant the little contradiction I pointed out above in this
thread.
But if you seriously ask in more general terms:
"32.5 Forecasts based on VAR models
To be written."
Remember the confusion at the Athens conference where people actually thought
it wouldn't be possible to do forecasts from VARs easily?
We should also document the --bootstrap option for ordered probit (and
perhaps extend it to ordinary probit)
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
10:30 a.m.
On Fri, 3 Aug 2018, Sven Schreiber wrote:
But if you seriously ask in more general terms:
"32.5 Forecasts based on VAR models
To be written."
Remember the confusion at the Athens conference where people actually thought
it wouldn't be possible to do forecasts from VARs easily?
Ok, I didn't write a proper subsection, but I wrote a little sample
script (with brief comments). I guess that's better than nothing.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
10:16 a.m.
Am 04.08.2018 um 12:08 schrieb Riccardo (Jack) Lucchetti:
On Fri, 3 Aug 2018, Sven Schreiber wrote:
> Am 03.08.2018 um 18:51 schrieb Riccardo (Jack) Lucchetti:
>> On Fri, 3 Aug 2018, Sven Schreiber wrote:
>>
>>> I'm also all for careful code reviews. Jack, in the meantime you
>>> could spend your energies on the documentation instead, hehe.
>>
> But if you seriously ask in more general terms:
> "32.5 Forecasts based on VAR models
> To be written."
> Remember the confusion at the Athens conference where people actually
> thought it wouldn't be possible to do forecasts from VARs easily?
We should also document the --bootstrap option for ordered probit (and
perhaps extend it to ordinary probit)
Yeah, well I know next to nothing about it...
cheers,
sven
10:17 a.m.
Am 04.08.2018 um 12:30 schrieb Riccardo (Jack) Lucchetti:
On Fri, 3 Aug 2018, Sven Schreiber wrote:
> But if you seriously ask in more general terms:
> "32.5 Forecasts based on VAR models
> To be written."
> Remember the confusion at the Athens conference where people actually
> thought it wouldn't be possible to do forecasts from VARs easily?
Ok, I didn't write a proper subsection, but I wrote a little sample
script (with brief comments). I guess that's better than nothing.
Absolutely.
-s
2080
days inactive
2096
days old
19 comments
3 participants
participants (3)
-
Allin Cottrell -
Riccardo (Jack) Lucchetti -
Sven Schreiber