From svetosch@gmx.net Thu May 22 04:55:20 2008
From: Sven Schreiber <svetosch@gmx.net>
To: gretl-devel@gretlml.univpm.it
Subject: [Gretl-devel] omit bug
Date: Thu, 22 May 2008 10:55:16 +0200
Message-ID: <483534F4.2080600@gmx.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1390501910069285987=="

--===============1390501910069285987==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Hi,
there seems to be a new bug introduced after 1.7.4; as always, I can 
provide more details later and/or file a bug report, but maybe this is 
already enough, and before I forget about it, here goes:

In a script of ours, we have an "omit" test statement after a standard 
(OLS) estimation, but in between there are some other calculations 
("scalar" and "matrix", so effectively "genr" I guess).

This worked fine up to 1.7.4, but current snapshots complain at the 
"omit" line, saying that no estimate has been performed so far.

cheers,
sven

--===============1390501910069285987==--


From cottrell@wfu.edu Thu May 22 09:07:04 2008
From: Allin Cottrell <cottrell@wfu.edu>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Thu, 22 May 2008 09:06:57 -0400
Message-ID: <alpine.LRH.1.10.0805220903550.435@ricardo.ecn.wfu.edu>
In-Reply-To: 483534F4.2080600@gmx.net
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0652253755469045852=="

--===============0652253755469045852==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Thu, 22 May 2008, Sven Schreiber wrote:

> there seems to be a new bug introduced after 1.7.4; as always, I 
> can provide more details later and/or file a bug report, but 
> maybe this is already enough, and before I forget about it, here 
> goes:
> 
> In a script of ours, we have an "omit" test statement after a 
> standard (OLS) estimation, but in between there are some other 
> calculations ("scalar" and "matrix", so effectively "genr" I 
> guess).

Please send a test case, I can't replicate this.  For example, the 
following works fine:

<script>
open data9-7
list xlist = 0 2 3 4
# see 'omit' at end
ols 1 xlist
matrix X = {xlist}
matrix XTXinv = inv(X'X)
scalar s2 = $sigma * $sigma
matrix V = s2*XTXinv
matrix se = sqrt(diag(V))
printf "\n%.12E\n", se
# 'omit' matches ols above
omit 3 4
</script>

Allin.

--===============0652253755469045852==--


From svetosch@gmx.net Fri May 23 08:57:49 2008
From: Sven Schreiber <svetosch@gmx.net>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Fri, 23 May 2008 14:57:46 +0200
Message-ID: <4836BF4A.5050903@gmx.net>
In-Reply-To: alpine.LRH.1.10.0805220903550.435@ricardo.ecn.wfu.edu
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============7116574274264790804=="

--===============7116574274264790804==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Am 22.05.2008 15:06, Allin Cottrell schrieb:
> On Thu, 22 May 2008, Sven Schreiber wrote:
> 
>> there seems to be a new bug introduced after 1.7.4; as always, I 
>> can provide more details later and/or file a bug report, but 
>> maybe this is already enough, and before I forget about it, here 
>> goes:
>>
>> In a script of ours, we have an "omit" test statement after a 
>> standard (OLS) estimation, but in between there are some other 
>> calculations ("scalar" and "matrix", so effectively "genr" I 
>> guess).
> 
> Please send a test case, I can't replicate this.  For example, the 
> following works fine:

Ok, it seems to be related to calling a user-defined function after 
estimation and before the restriction test; here is a demo script:

<script>
open denmark

function trivial(void)
result = 1
return scalar result
end function

ols LRM 0 LRY
scalar check = trivial()
omit LRY
</script>

This gives the mentioned error on today's CVS. I can't guarantee right 
now that this demo script works with 1.7.4, but our real-world script 
(where we noticed the bug) did.

In this case there is obviously an easy workaround (move the function 
call after the omit), but I can imagine cases where you want to use 
'restrict' with some value that you need to calculate in a function 
beforehand. So I'm inclined to think it's a bug.

thanks,
sven

--===============7116574274264790804==--


From cottrell@wfu.edu Fri May 23 14:50:10 2008
From: Allin Cottrell <cottrell@wfu.edu>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Fri, 23 May 2008 14:50:01 -0400
Message-ID: <alpine.LRH.1.10.0805231421450.2802@ricardo.ecn.wfu.edu>
In-Reply-To: 4836BF4A.5050903@gmx.net
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0838103206751096899=="

--===============0838103206751096899==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Fri, 23 May 2008, Sven Schreiber wrote:

> Am 22.05.2008 15:06, Allin Cottrell schrieb:
> > On Thu, 22 May 2008, Sven Schreiber wrote:
> > 
> > > In a script of ours, we have an "omit" test statement after 
> > > a standard (OLS) estimation, but in between there are some 
> > > other calculations ("scalar" and "matrix", so effectively 
> > > "genr" I guess).
>
> Ok, it seems to be related to calling a user-defined function 
> after estimation and before the restriction test; here is a demo 
> script:
> 
> <script>
> open denmark
> 
> function trivial(void)
> result = 1
> return scalar result
> end function
> 
> ols LRM 0 LRY
> scalar check = trivial()
> omit LRY
> </script>

Thanks, I now see (and remember!) what's happening. As you know, 
"omit" works on the "last model".  Sometime after the 1.7.4 
release I noticed that we had a problem with functions: if a 
function estimated a model, this would remain the "last model" on 
exit from the function, which is not what we want.

I then "fixed" this by simply NULLing out the last model on exit 
from a function, with the results you found.

I've now implemented the right fix, namely, when we enter a 
function we save a pointer to the current "last model" and on exit 
from a function we restore that pointer.

Should be OK in CVS and Windows snapshot.

Allin.

--===============0838103206751096899==--


From svetosch@gmx.net Fri May 23 16:27:37 2008
From: Sven Schreiber <svetosch@gmx.net>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Fri, 23 May 2008 22:27:33 +0200
Message-ID: <483728B5.9010201@gmx.net>
In-Reply-To: alpine.LRH.1.10.0805231421450.2802@ricardo.ecn.wfu.edu
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============6764016803251219405=="

--===============6764016803251219405==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Am 23.05.2008 20:50, Allin Cottrell schrieb:

> 
> I've now implemented the right fix, namely, when we enter a 
> function we save a pointer to the current "last model" and on exit 
> from a function we restore that pointer.
> 
> Should be OK in CVS and Windows snapshot.
> 

Yes it looks like it, thanks.

However, another new problem just appeared, the context is as follows:

First we do IV estimation:

tsls rpivstand endorhs d_predrhs  ; d_predrhs d1_wox --robust

Then we do various stuff:

 >accessing $coeff, $vcv>
<some 'printf'-ing, 'scalar', 'matrix' operations>
<a function call -- see the previous bug>
<some 'omit' tests>
<some 'restrict' tests>

and then we access the hausman test result and calculate a p-value by 
assuming one d.o.f. less than usual (don't ask why right now):

matrix hausmanresult = $hausman
pvalue X hausmanresult[1,2]-1 hausmanresult[1,1]

-- and this now gives an "invalid argument" error (it didn't in 1.7.4).

Note that the following reformulation of the last line provides a 
workaround, so it seems it has to do with "on-the-fly calculations" (if 
that's the right term):

temp=hausmanresult[1,2]-1
pvalue X temp hausmanresult[1,1]

BTW, on various occasions I noticed that arithmetic expressions cannot 
always be used in place of simple variables. But sometimes it does seem 
to work. Is there some rule when it should work, or is it sheer luck 
when it does?

Thanks,
Sven





--===============6764016803251219405==--


From cottrell@wfu.edu Fri May 23 16:44:31 2008
From: Allin Cottrell <cottrell@wfu.edu>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Fri, 23 May 2008 16:44:17 -0400
Message-ID: <alpine.LRH.1.10.0805231633170.2802@ricardo.ecn.wfu.edu>
In-Reply-To: 483728B5.9010201@gmx.net
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============2808010124942781362=="

--===============2808010124942781362==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Fri, 23 May 2008, Sven Schreiber wrote:

> Am 23.05.2008 20:50, Allin Cottrell schrieb:
> > Should be OK in CVS and Windows snapshot.
> > 
> 
> Yes it looks like it, thanks.

Good.

> However, another new problem just appeared...
> 
> matrix hausmanresult = $hausman
> pvalue X hausmanresult[1,2]-1 hausmanresult[1,1]
> 
> -- and this now gives an "invalid argument" error (it didn't in 
> 1.7.4)...

Oof, this is to do with a change I introduced at Stefano's 
behest: not "casting" 1x1 matrix results to scalars automatically.  
I'll have to think about that some more.

> BTW, on various occasions I noticed that arithmetic expressions 
> cannot always be used in place of simple variables. But 
> sometimes it does seem to work. Is there some rule when it 
> should work, or is it sheer luck when it does?

In general, arbitrary arithmetic should work for arguments to 
built-in functions (or user-defined functions for that matter),
while it's not very likely to work for arguments to straight
"commands".  However, in this particular case using the pvalue 
function won't help, right now, since it's a matrix/scalar thing.

I'll try to fix this properly for 1.7.5.

Allin.

--===============2808010124942781362==--


From cottrell@wfu.edu Fri May 23 17:34:35 2008
From: Allin Cottrell <cottrell@wfu.edu>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Fri, 23 May 2008 17:34:25 -0400
Message-ID: <alpine.LRH.1.10.0805231732280.15153@ricardo.ecn.wfu.edu>
In-Reply-To: alpine.LRH.1.10.0805231633170.2802@ricardo.ecn.wfu.edu
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============4483417877025720710=="

--===============4483417877025720710==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Fri, 23 May 2008, Allin Cottrell wrote:

> On Fri, 23 May 2008, Sven Schreiber wrote:
> > matrix hausmanresult = $hausman
> > pvalue X hausmanresult[1,2]-1 hausmanresult[1,1]
> > 
> > -- and this now gives an "invalid argument" error (it didn't in 
> > 1.7.4)...
> 
> Oof, this is to do with a change I introduced at Stefano's 
> behest: not "casting" 1x1 matrix results to scalars 
> automatically.  I'll have to think about that some more.

In CVS and the snapshot there's a fix which takes care of all the 
probability-related functions (including pvalue).  I'll search 
some more later for other functions that might be affected.

Allin.

--===============4483417877025720710==--


From r.lucchetti@univpm.it Fri May 23 17:53:20 2008
From: Riccardo (Jack) Lucchetti <r.lucchetti@univpm.it>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Fri, 23 May 2008 23:53:18 +0200
Message-ID: <alpine.DEB.1.10.0805232344470.29575@ec-4.econ.univpm.it>
In-Reply-To: alpine.LRH.1.10.0805231633170.2802@ricardo.ecn.wfu.edu
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============6991879335507862182=="

--===============6991879335507862182==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit

On Fri, 23 May 2008, Allin Cottrell wrote:

> Oof, this is to do with a change I introduced at Stefano's
> behest: not "casting" 1x1 matrix results to scalars automatically.
> I'll have to think about that some more.

I can see Stefano's point, but there are also advantages in auto-casting a 
1x1 matrix to a scalar. By the way (Stefano, are you there?) if you really 
need to ensure that the result of an operation is a matrix, you can always 
force it via the "matrix" alias, as in

matrix x = foo(z)

So, in my view, there are pros and cons to either choice. I think we ought 
to reach a collective decision here. So my proposal is: everyone 
interested, please have your say now. (Myself, I'd say "auto-cast".) 
Allin, just wait for a few days and then whatever you decide is the law.


Riccardo (Jack) Lucchetti
Dipartimento di Economia
Università Politecnica delle Marche

r.lucchetti(a)univpm.it
http://www.econ.univpm.it/lucchetti

--===============6991879335507862182==--


From svetosch@gmx.net Sat May 24 11:00:45 2008
From: Sven Schreiber <svetosch@gmx.net>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Sat, 24 May 2008 17:00:43 +0200
Message-ID: <48382D9B.6010307@gmx.net>
In-Reply-To: alpine.LRH.1.10.0805231633170.2802@ricardo.ecn.wfu.edu
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============6061264641399546058=="

--===============6061264641399546058==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Am 23.05.2008 22:44, Allin Cottrell schrieb:
> On Fri, 23 May 2008, Sven Schreiber wrote:
> 

>> However, another new problem just appeared...
>>
>> matrix hausmanresult = $hausman
>> pvalue X hausmanresult[1,2]-1 hausmanresult[1,1]
>>
>> -- and this now gives an "invalid argument" error (it didn't in 
>> 1.7.4)...
> 
> Oof, this is to do with a change I introduced at Stefano's 
> behest: not "casting" 1x1 matrix results to scalars automatically.  
> I'll have to think about that some more.

first, one thing I don't understand here: the second term 
(hausmanresult[1,1]) doesn't seem problematic (see the workaround I 
mentioned, where it is unchanged). But that would mean it is treated as 
a scalar, no? Then why does the first term (hausmanresult[1,2]) give 
something different?

secondly, I think there are two different issues involved here:
1) The first one is what data type is returned by a function (built-in 
or user-defined).
2) The second issue is what happens if you pick elements from a matrix, 
as in the above example.

 From my experience with earlier discussions about those issues on the 
numpy (numerical python) mailing list, these issues can and should be 
decided separately.

With respect to the first issue, I have some sympathy with Stefano's 
view that the returned datatype should not depend on the specific matrix 
dimension in particular cases.

As for the second issue, I would say that explicitly indexing a single 
scalar element from a matrix --as in the above example-- should return a 
scalar. However, that would *not* mean that each indexing range which 
refers to a 1x1 submatrix should be returned as a scalar. For example, 
let's say somebody picks the first row of a matrix mymat by using 
mymat[1,] in an expression. In general the first row would not be just a 
single element, and as such mymat[1,] should always be a matrix, even if 
it turns out to be just a 1x1 matrix.


[and now continuing a different topic:]

> 
>> BTW, on various occasions I noticed that arithmetic expressions 
>> cannot always be used in place of simple variables. But 
>> sometimes it does seem to work. Is there some rule when it 
>> should work, or is it sheer luck when it does?
> 
> In general, arbitrary arithmetic should work for arguments to 
> built-in functions (or user-defined functions for that matter),
> while it's not very likely to work for arguments to straight
> "commands".  

In principle I think I understand, but I'm not sure the demarcation 
between commands and functions is very clear in gretl. What would that 
'pvalue' thing be for example?

-sven

--===============6061264641399546058==--


From cottrell@wfu.edu Sat May 24 15:15:50 2008
From: Allin Cottrell <cottrell@wfu.edu>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Sat, 24 May 2008 15:15:39 -0400
Message-ID: <alpine.LRH.1.10.0805241503340.2831@ricardo.ecn.wfu.edu>
In-Reply-To: 48382D9B.6010307@gmx.net
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============3814209884889667614=="

--===============3814209884889667614==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Sat, 24 May 2008, Sven Schreiber wrote:

[ first, about cating 1x1 matrices to scalars, to which I'll reply 
later, then... ]

> > > BTW, on various occasions I noticed that arithmetic 
> > > expressions cannot always be used in place of simple 
> > > variables. But sometimes it does seem to work. Is there some 
> > > rule when it should work, or is it sheer luck when it does?
> > 
> > In general, arbitrary arithmetic should work for arguments to 
> > built-in functions (or user-defined functions for that 
> > matter), while it's not very likely to work for arguments to 
> > straight "commands".
> 
> In principle I think I understand, but I'm not sure the 
> demarcation between commands and functions is very clear in 
> gretl. What would that 'pvalue' thing be for example?

For the most part the distinction is pretty clear: a function is 
something where all the arguments are wrapped in parentheses and 
separated by commas, and a command is everything else.  But there 
are a few hybrid cases, and "pvalue" is one such.  Since we have a 
perfectly good pvalue function with the same semantics, we 
implement the "command" form by composing and executing a call to 
"genr"; this gives more flexibility on the form of the arguments.

Allin.

--===============3814209884889667614==--


From svetosch@gmx.net Fri May 30 10:37:01 2008
From: Sven Schreiber <svetosch@gmx.net>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Fri, 30 May 2008 16:38:26 +0200
Message-ID: <48401162.3030209@gmx.net>
In-Reply-To: alpine.LRH.1.10.0805241503340.2831@ricardo.ecn.wfu.edu
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============7722510469355564076=="

--===============7722510469355564076==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Allin Cottrell schrieb:
> On Sat, 24 May 2008, Sven Schreiber wrote:
> 
> [ first, about cating 1x1 matrices to scalars, to which I'll reply 
> later, then... ]
> 

any news here about the 1x1 matrix treatment?

cheers,
sven


--===============7722510469355564076==--


From cottrell@wfu.edu Fri May 30 14:32:29 2008
From: Allin Cottrell <cottrell@wfu.edu>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] omit bug
Date: Fri, 30 May 2008 14:32:14 -0400
Message-ID: <alpine.LRH.1.10.0805301424580.1768@ricardo.ecn.wfu.edu>
In-Reply-To: 48401162.3030209@gmx.net
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0261546388969210141=="

--===============0261546388969210141==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Fri, 30 May 2008, Sven Schreiber wrote:

> Allin Cottrell schrieb:
> > On Sat, 24 May 2008, Sven Schreiber wrote:
> > 
> > [ first, about cating 1x1 matrices to scalars, to which I'll reply later,
> > then... ]
> > 
> any news here about the 1x1 matrix treatment?

The current status is:

* If a genr expression "naturally" produces a matrix result, but 
the matrix is 1x1, and the user has not forced the issue by 
specifying a result-type of "scalar" or "matrix", then we leave 
the result as a 1x1 matrix.  (As per Stefano's suggestion.)

* But we've tried to make inter-conversion between 1x1 matrices 
and scalars (inside genr) more flexible, so that the type of
the result should make less difference to the user.

Allin.



--===============0261546388969210141==--