12:31 p.m.
Sorry for the flood of messages from me, I'm doing some "heavy
scripting" ATM and so I stumble over a lot of stuff...
The example script below prints two identical series under the name of
"s1", which is "obviously" wrong, although I'm not sure which
would be
the correct behavior -- I can think of two correct variants:
1) "s1" and "s2" are in the namespace of test() and thus should not be
accessible at all, so the script should raise an error. (Although then
it's not quite clear how to access the members of the returned list.)
or
2) "s1" should be properly updated when it is changed inside the
function and then returned as a member of the list "myl".
This is on today's version from CVS.
thanks,
sven
<demo>
function test(void)
series s1 = randgen(u,0,1)
series s2 = randgen(u,2,3)
list myl = s1 s2
return list myl
end function
loop 2
list check = test()
print s1
end loop
</demo>
10:53 a.m.
On Sat, 9 Feb 2008, Sven Schreiber wrote:
The example script below prints two identical series under the
name of "s1", which is "obviously" wrong, although I'm not sure
which would be the correct behavior -- I can think of two
correct variants:
1) "s1" and "s2" are in the namespace of test() and thus should
not be accessible at all, so the script should raise an error.
(Although then it's not quite clear how to access the members of
the returned list.)
No, s1 and s2 are moved to the namespace of the caller when the
caller make use of (assigns) the list return. That's by design.
2) "s1" should be properly updated when it is changed
inside the
function and then returned as a member of the list "myl".
Yes. There's something wrong here. I notice that it works as
expected if you give the list itself as an argument to "print":
function test(void)
series s1 = randgen(u,0,1)
series s2 = randgen(u,2,3)
list myl = s1 s2
return list myl
end function
nulldata 10
loop 2
list check = test()
print check -o
end loop
I'll try to work out what's happening when you print "s1" rather
than "check".
Allin
11:04 a.m.
On Sat, 9 Feb 2008, Sven Schreiber wrote:
The example script below prints two identical series under the
name of "s1", which is "obviously" wrong...
Ah, there's a loose end here, in terms of behavior when a function
returns a list: what should happen if the list contains
one or more variables that have the same name as a variable that
already exists at the caller level?
What's happening at present can be exposed by this variant on
Sven's script:
function test(void)
series s1 = randgen(u,0,1)
series s2 = randgen(u,2,3)
list myl = s1 s2
print "in test"
print s1 -o
return list myl
end function
nulldata 4
loop 2
list check = test()
varlist
print "in caller"
print s1 -o
end loop
The last "varlist" produces:
Listing 6 variables:
0) const 1) index 2) s1 3) s2 4) s1
5) s2
which is not nice. It seems to me the second iteration should
overwrite s1 and s2 -- agreed?
Allin.
12:04 p.m.
Am 10.02.2008 17:04, Allin Cottrell schrieb:
On Sat, 9 Feb 2008, Sven Schreiber wrote:
> The example script below prints two identical series under the
> name of "s1", which is "obviously" wrong...
Ah, there's a loose end here, in terms of behavior when a function
returns a list: what should happen if the list contains
one or more variables that have the same name as a variable that
already exists at the caller level?
What's happening at present can be exposed by this variant on
Sven's script:
function test(void)
series s1 = randgen(u,0,1)
series s2 = randgen(u,2,3)
list myl = s1 s2
print "in test"
print s1 -o
return list myl
end function
nulldata 4
loop 2
list check = test()
varlist
print "in caller"
print s1 -o
end loop
The last "varlist" produces:
Listing 6 variables:
0) const 1) index 2) s1 3) s2 4) s1
5) s2
which is not nice. It seems to me the second iteration should
overwrite s1 and s2 -- agreed?
At first I thought "yes" (and it was what I was proposing originally),
but then it occurred to me that the fact that returning lists from
functions would affect the caller's namespaces is a "policy breach".
Because in theory it's possible that at the caller level there exists a
series of the same name just by coincidence, which would then get
deleted. So before you could use a function written by somebody else you
would have to study it and make sure it doesn't have this side effect.
IMHO that's bad.
Ok, so the probability of this problem occurring in practice may be
minimal, and so feel free to discard this objection.
But without having thought too much about it, let me just give an
example of how it could perhaps work:
Suppose that "list indexing" is introduced as a new syntax; that would
mean that in the above example you could access the series within the
list 'check' by a one-dimensional index: check[1], check[2] etc. up to
check[nelem(check)]. (And you would not be able to use the names 's1' or
's2' at the caller level, they would be restricted to the
scope/namespace of the function.) However, I admit it's not 100% clear
how the series would be displayed in the GUI without running into the
namespace collision problem again.
just a thought,
sven
2:35 p.m.
On Sun, 10 Feb 2008, Sven Schreiber wrote:
Am 10.02.2008 17:04, Allin Cottrell schrieb:
> Ah, there's a loose end here, in terms of behavior when a
> function returns a list: what should happen if the list
> contains one or more variables that have the same name as a
> variable that already exists at the caller level? ...
[ and went on to suggest that in this case the original values
of the variables in question should be overwritten ].
At first I thought "yes" (and it was what I was proposing
originally), but then it occurred to me that the fact that
returning lists from functions would affect the caller's
namespaces is a "policy breach". Because in theory it's possible
that at the caller level there exists a series of the same name
just by coincidence, which would then get deleted.
Yes, I agree that's a possible issue.
I was going to say there's a precedent for this sort of thing with
various built-in commands/functions -- such as logs, lags, and
diff -- which generate automatically-named variables. However, I
notice that at some point we adopted the policy of avoiding name
collision by mangling the names of the generated series if
necessary.
<script>
open data4-10
series l_ENROLL = 10
logs ENROLL
print l_EN*
</script>
After this we have l_ENROLL, which still equals 10, and l_ENROLa,
which holds the log of ENROLL.
We could try something similar with list-returning user functions,
but actually on second thoughts I'm not so sure this is the right
thing to do. I'm inclined to say that so long as the names used
by the function are clearly stated in its documentation -- so the
user can take evasive action if necessary -- the cleaner policy is
to overwrite.
I was a bit surprised by the behavior of "logs" noted above, and
I'm not sure it really conforms to intent; as I recall, the
primary intent behind name-mangling was to ensure uniqueness
in case the names of the source variables were too long. E.g.
logs longnameseries1 longnameseries2
where something has to be done or we'll get 2 variables named
l_longnameserie.
For now I have it that list-returning user functions just
overwrite, but perhaps we need to discuss this some more.
Allin.
4:12 p.m.
On Mon, 11 Feb 2008, Allin Cottrell wrote:
For now I have it that list-returning user functions just
overwrite, but perhaps we need to discuss this some more.
I'm personally in favour of the overwrite policy. The logs command
generates variables with the prefix "l_" plus the original name. This is
consistent, logical and predictable. Imagine this scenario: you have a
huge dataset with hundreds of variables, you take logs _en masse_ via the
nice and convenient "logs" command, only to find, dozens of lines later,
that what you thought was under the name "l_varname34" doesn't contain
what you want, because it's under "l_varname34a". Very irritating, and
hellish to debug. The only good thing that may come out of this situation
is the invention, if you're creative, of swearwords previously unknown to
man.
Riccardo (Jack) Lucchetti
Dipartimento di Economia
Università Politecnica delle Marche
r.lucchetti(a)univpm.it
http://www.econ.univpm.it/lucchetti
4:59 p.m.
On Mon, 11 Feb 2008, Riccardo (Jack) Lucchetti wrote:
On Mon, 11 Feb 2008, Allin Cottrell wrote:
> For now I have it that list-returning user functions just
> overwrite, but perhaps we need to discuss this some more.
I'm personally in favour of the overwrite policy. The logs
command generates variables with the prefix "l_" plus the
original name. This is consistent, logical and predictable.
Imagine this scenario: you have a huge dataset with hundreds of
variables, you take logs _en masse_ via the nice and convenient
"logs" command, only to find, dozens of lines later, that what
you thought was under the name "l_varname34" doesn't contain
what you want, because it's under "l_varname34a". Very
irritating, and hellish to debug. The only good thing that may
come out of this situation is the invention, if you're creative,
of swearwords previously unknown to man.
Yes, that's the viewing I'm coming round to ;-).
There's still an issue to worry about with long variable names.
Obviously we want to ensure that, e.g.,
logs xlist
returns uniquely named variables. And that's not too hard (though
making the unique names "nice" in all cases is another matter).
We can overwrite any pre-existing variables, but mangle to avoid
naming conflicts among the variables that are *generated by this
particular command*.
But what about successive invocations:
logs xlist
logs zlist
where xlist and zlist contain long names that will yield
"cross-conflicting" log names, so to speak? Under the new
proposal as it stands, some of the logs created by the first
command would be overwritten (I think).
But maybe worrying about this falls into the category of trying a
little too hard to stop users from shooting themselves in the
foot.
We can ensure the docs are clear on this point, and leave it up to
the user to use variable names of a sane length. Maybe also print
a message when an existing variable is redefined via an
auto-naming command/function.
Allin.
5:26 p.m.
On Mon, 11 Feb 2008, Allin Cottrell wrote:
But what about successive invocations:
logs xlist
logs zlist
where xlist and zlist contain long names that will yield
"cross-conflicting" log names, so to speak? Under the new
proposal as it stands, some of the logs created by the first
command would be overwritten (I think).
And rightly so, if you ask me. Now, we may consider raising VNAMELEN to
24 or 32 to minimise the chances of this happening, but then formatting
the output of estimation commands becomes a problem. The Stata guys
solved this issue by truncating variable names on output, using
reasonable conventions. Example:
. regress very_long_identifier_y very_long_identifier1 very_long_identifier2
Source | SS df MS Number of obs = 753
-------------+------------------------------ F( 2, 750) = 45.82
Model | 860.569119 2 430.284559 Prob > F = 0.0000
Residual | 7042.54159 750 9.39005546 R-squared = 0.1089
-------------+------------------------------ Adj R-squared = 0.1065
Total | 7903.11071 752 10.5094557 Root MSE = 3.0643
------------------------------------------------------------------------------
very_long_~y | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
very_long~_1 | -.1177097 .0467719 -2.52 0.012 -.209529 -.0258903
very_long_~2 | .5480657 .061962 8.85 0.000 .4264261 .6697053
_cons | -2.889082 .623721 -4.63 0.000 -4.113529 -1.664636
------------------------------------------------------------------------------
Perhaps we could do the same.
But maybe worrying about this falls into the category of trying a
little too hard to stop users from shooting themselves in the
foot.
We can ensure the docs are clear on this point, and leave it up to
the user to use variable names of a sane length. Maybe also print
a message when an existing variable is redefined via an
auto-naming command/function.
I heartily agree.
Riccardo (Jack) Lucchetti
Dipartimento di Economia
Università Politecnica delle Marche
r.lucchetti(a)univpm.it
http://www.econ.univpm.it/lucchetti
4:59 a.m.
Am 11.02.2008 22:12, Riccardo (Jack) Lucchetti schrieb:
On Mon, 11 Feb 2008, Allin Cottrell wrote:
> For now I have it that list-returning user functions just
> overwrite, but perhaps we need to discuss this some more.
I'm personally in favour of the overwrite policy. The logs command
generates variables with the prefix "l_" plus the original name. This is
consistent, logical and predictable. Imagine this scenario: you have a
huge dataset with hundreds of variables, you take logs _en masse_ via
the nice and convenient "logs" command, only to find, dozens of lines
later, that what you thought was under the name "l_varname34" doesn't
contain what you want, because it's under "l_varname34a". Very
irritating, and hellish to debug. The only good thing that may come out
of this situation is the invention, if you're creative, of swearwords
previously unknown to man.
Yeah I see your point, but what about the swearwords of those who lose
their existing variable that was hand-typed with 15 million
observations? (Slight exaggeration to make my point...)
I have thought about it a little bit but could only think of a solution
for the builtin functions like the ones you're talking about: names
starting with 'l_' and the like could be made "semi-reserved" in the
manual, explaining that you risk losing them.
But IMHO the situation is totally different for user-defined functions:
users should not be required to do a risk analysis or thoroughly study
the documentation of a function before playing around with it.
Otherwise, gretl's new and great function package approach would be
unsafe by design and it would ultimately be doomed IMHO. I see no
solution for this problem except to disable the mentioned "namespace
breach". (Think of how likely it is to have series named 'y' or 'u'
or
'x' and a function author choosing those names as well. It seems to me
that the clashes I'm talking about are not just theoretical
possibilities.)
One issue is very important IMHO: dataloss by design should be
unacceptable for gretl, no matter how convenient it would be for savvy
users (or developers)!
thanks,
sven
1:52 p.m.
On Tue, 12 Feb 2008, Sven Schreiber wrote:
I have thought about it a little bit but could only think of a
solution for the builtin functions like the ones you're talking
about: names starting with 'l_' and the like could be made
"semi-reserved" in the manual, explaining that you risk losing
them.
But IMHO the situation is totally different for user-defined
functions: users should not be required to do a risk analysis or
thoroughly study the documentation of a function before playing
around with it.
Just thinking out loud, but here's one possibility: whenever a
function returns a list (I think that's the only case we're
talking about) there's an automatically accepted final string
argument, namely the "basename" for variables in the returned
list, which then (say) get consecutive numeric suffixes.
For example, consider
function foo (series x, int opt)
...
return list xlist
end function
If you called foo as
list mylist = foo(myvar, 4, "myname")
then gretl would intercept the list at return time and name its
members as myname1, myname2 and so on, before they're shifted into
caller space.
One could get fancier by allowing a more flexible naming template
as the extra argument, rather than just a prefix, but that's the
basic idea.
Allin.
11:55 a.m.
New subject: list of lags
There is a backward compatibility problem with the recent changes in
the "list" command. Until now there were no problem with this command
list xlist = Ct Ct(-1)
the contemporaneous value of Ct and the first lag were added to the list.
But now I obtain the error
"Data types not conformable for operation"
As a workaround I can use
list laglist = lags(1, Ct)
list xlist = Ct laglist
obtaining the same result, but I think the old syntax should be correct now as
well.
--
Ignacio Diaz-Emparanza
DEPARTAMENTO DE ECONOMÍA APLICADA III (ECONOMETRÍA Y ESTADÍSTICA)
UPV/EHU
Avda. Lehendakari Aguirre, 83 | 48015 BILBAO
T.: +34 946013732 | F.: +34 946013754
www.et.bs.ehu.es
8:17 p.m.
New subject: list of lags
On Thu, 14 Feb 2008, Ignacio Diaz-Emparanza wrote:
There is a backward compatibility problem with the recent
changes in the "list" command. Until now there were no problem
with this command
list xlist = Ct Ct(-1)
the contemporaneous value of Ct and the first lag were added to
the list. But now I obtain the error
"Data types not conformable for operation"
I believe that syntax should now be OK in CVS. Please come back
if it's not working right.
Allin.
1:50 a.m.
New subject: list of lags
On Thu, 14 Feb 2008, Allin Cottrell wrote:
On Thu, 14 Feb 2008, Ignacio Diaz-Emparanza wrote:
> There is a backward compatibility problem with the recent
> changes in the "list" command. Until now there were no problem
> with this command
>
> list xlist = Ct Ct(-1)
>
> the contemporaneous value of Ct and the first lag were added to
> the list. But now I obtain the error
>
> "Data types not conformable for operation"
I believe that syntax should now be OK in CVS. Please come back
if it's not working right.
I don't remember if the syntax
list X = x(-1 to -4)
ever worked (my guess is yes, but I may be wrong). It certainly doesn't
now; should it?
Riccardo (Jack) Lucchetti
Dipartimento di Economia
Università Politecnica delle Marche
r.lucchetti(a)univpm.it
http://www.econ.univpm.it/lucchetti
8:24 a.m.
New subject: list of lags
On Fri, 15 Feb 2008, Riccardo (Jack) Lucchetti wrote:
I don't remember if the syntax
list X = x(-1 to -4)
ever worked (my guess is yes, but I may be wrong). It certainly
doesn't now; should it?
At present it works in the context of a regression specification
ols y 0 x(-1 to -4)
but not in stand-alone list creation. I suppose it could be
enabled, though it would be fiddly (" to " is obviously ad hoc,
from the genr point of view).
Allin.
1:16 p.m.
New subject: list of lags (again)
On Fri, 15 Feb 2008, Allin Cottrell wrote:
On Fri, 15 Feb 2008, Riccardo (Jack) Lucchetti wrote:
> I don't remember if the syntax
>
> list X = x(-1 to -4)
>
> ever worked (my guess is yes, but I may be wrong). It certainly
> doesn't now; should it?
It should work now in CVS.
I've enabled a couple more more things in relation to lists:
list L = x1 x2 time
("time" does not have to exist already)
list xlist = x1 x2 x3
list L = xpx(xlist)
(L holds squares and cross-products)
What I'm working towards here is the possibility of trashing the
list-parser that still exists in interact.c, for lists provided in
the context of, e.g., estimation commands. I'd like to hand it
all over to genr at some point.
Allin.
1:40 p.m.
New subject: list of lags (again)
On Fri, 15 Feb 2008, Allin Cottrell wrote:
I've enabled a couple more more things in relation to lists:
list L = x1 x2 time
("time" does not have to exist already)
list xlist = x1 x2 x3
list L = xpx(xlist)
(L holds squares and cross-products)
What I'm working towards here is the possibility of trashing the
list-parser that still exists in interact.c, for lists provided in
the context of, e.g., estimation commands. I'd like to hand it
all over to genr at some point.
This is all excellent, and certainly improves scripting a lot.
I'm wondering if lists are now mature enough to show up in the GUI: at
this point, someone who only uses the GUI may not even know what a list
is, which is IMO a shame. Some random thoughts follow: it'd be nice to
* highlight one or more variables from the main window, and have "Define
as list" in the right-click menu; a similar item could also go in the
"Add" menu;
* show named lists in the estimation windows, in a separate pane somewhere
(under the variables, maybe?);
* make defined lists visible, maybe together with variables, like Eviews
does for "groups", or in the session icon view.
Other ideas?
Riccardo (Jack) Lucchetti
Dipartimento di Economia
Università Politecnica delle Marche
r.lucchetti(a)univpm.it
http://www.econ.univpm.it/lucchetti
6120
days inactive
6126
days old
16 comments
4 participants
participants (4)
-
Allin Cottrell -
Ignacio Diaz-Emparanza -
Riccardo (Jack) Lucchetti -
Sven Schreiber