From r.lucchetti@univpm.it Fri Jul 18 08:13:03 2008
From: Riccardo (Jack) Lucchetti <r.lucchetti@univpm.it>
To: gretl-devel@gretlml.univpm.it
Subject: [Gretl-devel] PROTECT_LIST: first casualties (long)
Date: Fri, 18 Jul 2008 14:13:00 +0200
Message-ID: <alpine.DEB.1.10.0807181256280.29560@ec-4.econ.univpm.it>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============7781227355565939047=="

--===============7781227355565939047==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit

Consider the following script:

<script>
set echo off
set messages off

function foo(list X)
     printf "\nInside function foo:\n"
     loop foreach i X -q
         printf "$i\n"
     end loop
end function

function bar(list X)
     printf "\nInside function bar:\n"
     loop foreach i X -q
         printf "%s\n", varname(i)
     end loop
end function

nulldata 10
x1 = normal()
x2 = normal()
list X = const x1 x2
printf "\nOutside a function:\n"
loop foreach i X -q
     printf "$i\n"
end loop

foo(X)
bar(X)
</script>

With current CVS, with PROTECT_LISTS=1, it yields:

<output>
Outside a function:
const
x1
x2

Inside function foo:
const
2
3

Inside function bar:
index
x1
x2
</output>

There are several points I'd like to raise: bear with me, this is going to 
be neither short nor painless.

Looking at the output of "foo", clearly the constant is treated 
specially, which should not happen. This also affects the output of "bar", 
in which varname(i) becomes varname(const)=varname(1)="index"

Even if the above particular asymmetry is removed, another one becomes 
evident: foreach loops inside functions behave differently from those 
outside. In the latter case, "$i" yields a string; in the former, an 
integer. This has all sort of weird consequences: for example, a statement 
like "genr $i_xxx = normal()" is legal outside a function but becomes 
illegal inside, since identifiers cannot start with numbers.

A possible way out would be to endow variables passed through lists with 
local aliases, like for example "X->const", or "X.x1", in which we use 
some "illegal" marker ("->" or "." in the above examples) to make sure 
that no name clashes inside functions can occur.

This isn't trivial to implement, and has the added disadvantage that you 
may end up with monsters like "LongNameList.VeryVerboseVariable", but the 
alternatives are not very palatable either: with the new mechanism (which, 
I repeat, I consider indispensable), either we're prepared to live with a 
syntax inconsistency (terrible) or we change the meaning of $i in foreach 
loops outside functions. This would be a huge backward-incompatible change 
(are you there, Sven?). In other words, if the "local name" solution (plan 
A) proves impractical, I propose to extend the varname() mechanism for 
retrieving variable names to all foreach loops (plan B). The number of 
existing scripts that will be broken if plan B is carried forward is 
doubtlessly enormous, but I think it would be a sad necessity.

Now, the question is: what should our policy be?

Riccardo (Jack) Lucchetti
Dipartimento di Economia
Università Politecnica delle Marche

r.lucchetti(a)univpm.it
http://www.econ.univpm.it/lucchetti

--===============7781227355565939047==--


From cottrell@wfu.edu Sat Jul 19 07:08:16 2008
From: Allin Cottrell <cottrell@wfu.edu>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] PROTECT_LIST: first casualties (long)
Date: Sat, 19 Jul 2008 07:08:13 -0400
Message-ID: <alpine.LRH.1.10.0807190659240.1767@ricardo.ecn.wfu.edu>
In-Reply-To: alpine.DEB.1.10.0807181256280.29560@ec-4.econ.univpm.it
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============3481547459900237744=="

--===============3481547459900237744==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Fri, 18 Jul 2008, Riccardo (Jack) Lucchetti wrote:

[long quote needed for context]

> Consider the following script:
> 
> <script>
> set echo off
> set messages off
> 
> function foo(list X)
>     printf "\nInside function foo:\n"
>     loop foreach i X -q
>         printf "$i\n"
>     end loop
> end function
> 
> function bar(list X)
>     printf "\nInside function bar:\n"
>     loop foreach i X -q
>         printf "%s\n", varname(i)
>     end loop
> end function
> 
> nulldata 10
> x1 = normal()
> x2 = normal()
> list X = const x1 x2
> printf "\nOutside a function:\n"
> loop foreach i X -q
>     printf "$i\n"
> end loop
> 
> foo(X)
> bar(X)
> </script>
> 
> With current CVS, with PROTECT_LISTS=1, it yields:
> 
> <output>
> Outside a function:
> const
> x1
> x2
> 
> Inside function foo:
> const
> 2
> 3
> 
> Inside function bar:
> index
> x1
> x2
> </output>
> 
> Looking at the output of "foo", clearly the constant is treated 
> specially, which should not happen. ...

Agreed.  This is now fixed in CVS.

> This also affects the output of "bar", in which varname(i) 
> becomes varname(const)=varname(1)="index"

No.  The fact that you get "index" as the output for the first 
iteration in the "bar" loop is the result of using "i" rather than 
"$i".  The output is correct: i always equals 1 on the first 
iteration, and variable 1 is called "index".

> Even if the above particular asymmetry is removed, another one 
> becomes evident: foreach loops inside functions behave 
> differently from those outside...

The way PROTECT_LISTS works at present, the key point is not 
whether we're inside a function as such, but whether the list over 
which we're looping, using "foreach", has been made available as a 
function argument, or is "native" to the current level of function 
execution.

I think it's inevitable that there has to be some such difference.  
I agree that we need to think some more about what's the least 
confusing way of handling the issue.

Allin.

--===============3481547459900237744==--


From r.lucchetti@univpm.it Sat Jul 19 09:27:47 2008
From: Riccardo (Jack) Lucchetti <r.lucchetti@univpm.it>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] PROTECT_LIST: first casualties (long)
Date: Sat, 19 Jul 2008 15:27:41 +0200
Message-ID: <alpine.DEB.1.10.0807191510300.19457@ec-4.econ.univpm.it>
In-Reply-To: alpine.LRH.1.10.0807190659240.1767@ricardo.ecn.wfu.edu
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============4073547980884832681=="

--===============4073547980884832681==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit

On Sat, 19 Jul 2008, Allin Cottrell wrote:

> The way PROTECT_LISTS works at present, the key point is not
> whether we're inside a function as such, but whether the list over
> which we're looping, using "foreach", has been made available as a
> function argument, or is "native" to the current level of function
> execution.

Ah, ok, this is clear to me now, thanks.

> I think it's inevitable that there has to be some such difference.
> I agree that we need to think some more about what's the least
> confusing way of handling the issue.

Would I be wrong if I summarised the current state of affairs (with 
PROTECT_LISTS=1) as follows?

List is local: $i returns a string and $i_foo is a valid identifier and

 	genr $i_foo = normal()

is a valid statement.

List comes "from the outside" as a function argument: $i returns a number 
and $i_foo is _not_ a valid identifier. To get the same effect as above, 
you need to work with strings, as in

 	sprintf vnam "%s_foo", varname($i)
 	genr @vnam = normal()



Riccardo (Jack) Lucchetti
Dipartimento di Economia
Università Politecnica delle Marche

r.lucchetti(a)univpm.it
http://www.econ.univpm.it/lucchetti

--===============4073547980884832681==--


From cottrell@wfu.edu Sun Jul 20 08:32:58 2008
From: Allin Cottrell <cottrell@wfu.edu>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] PROTECT_LIST: first casualties (long)
Date: Sun, 20 Jul 2008 08:32:49 -0400
Message-ID: <alpine.LRH.1.10.0807200817100.14537@ricardo.ecn.wfu.edu>
In-Reply-To: alpine.DEB.1.10.0807191510300.19457@ec-4.econ.univpm.it
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============2406754917173537016=="

--===============2406754917173537016==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Sat, 19 Jul 2008, Riccardo (Jack) Lucchetti wrote:

> Would I be wrong if I summarised the current state of affairs 
> (with PROTECT_LISTS=1) as follows?
> 
> List is local: $i returns a string and $i_foo is a valid 
> identifier and
> 
> 	genr $i_foo = normal()
> 
> is a valid statement.
> 
> List comes "from the outside" as a function argument: $i returns 
> a number and $i_foo is _not_ a valid identifier. To get the same 
> effect as above, you need to work with strings, as in
> 
> 	sprintf vnam "%s_foo", varname($i)
> 	genr @vnam = normal()

As of the time of writing, this is quite right!  However, in 
current CVS I'm experimenting with the alternative whereby you can 
refer to list-variables using listname.varname.  For example, in 
this scheme the cubes function would look like this:

function make_cubes (list xlist)
   list retlist = null
   loop foreach i xlist
      sprintf newname "%14s_3", "$i"
      series @newname = xlist.$i^3
      setinfo @newname -d "cube of $i"
      list retlist += @newname
    end loop
    return list retlist
end function

That is, "$i" gets a variable-name string as usual, but if you 
want to use this to get the values of a listed variable (where the 
list has been supplied as an argument) you need the "listname." 
prefix.

I'm trying this because I accept your point that it's very 
much preferable that "$i" does the same thing in all foreach-loop 
contexts if this is at all feasible.

Limited testing suggests that listname.varname works pretty well.  
But it has this limitation: it will be unpacked properly only in a 
genr-type context.  Per contra, if you try, e.g.

   loop foreach i xlist
      print xlist.$i
   end loop

"xlist.<name>" will not be recognized as a valid construction.  
Now obviously the above fragment is artificial and redundant, 
since you could just do

   print xlist

I'm not sure yet how likely it is that a real function would run 
into this sort of problem.  If it is a real issue, I think the 
best solution is to pass all list-construction through genr, which 
is something I've had in mind for a while anyway.

Allin.


--===============2406754917173537016==--


From cottrell@wfu.edu Sun Jul 20 10:29:38 2008
From: Allin Cottrell <cottrell@wfu.edu>
To: gretl-devel@gretlml.univpm.it
Subject: Re: [Gretl-devel] PROTECT_LIST: first casualties (long)
Date: Sun, 20 Jul 2008 10:29:35 -0400
Message-ID: <alpine.LRH.1.10.0807201027080.29658@ricardo.ecn.wfu.edu>
In-Reply-To: alpine.LRH.1.10.0807200817100.14537@ricardo.ecn.wfu.edu
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============6414449054839032875=="

--===============6414449054839032875==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Sun, 20 Jul 2008, Allin Cottrell wrote:

> Limited testing suggests that listname.varname works pretty well.  
> But it has this limitation: it will be unpacked properly only in a 
> genr-type context...

That is no longer true.  That is, recognition of "list.var" is 
now extended to more contexts.  But it still won't work in all 
contexts.

Allin.

--===============6414449054839032875==--