1:11 p.m.
Hi all,
I was revisiting some code I wrote a few years ago, and came across what is at least
changed (and unexpected) behavior. I have a script that loops over various numbers of
possible structural breaks, and does some MCMC sampling for each value. I want to write
the mcmc draws to separate data sets labeled by the number of breaks.
Here's what I originally had:
<hansl>
loop ii=1..breaks --quiet
Nbreaks = ii-1
# do a bunch of stuff
loop draws = 1..ndraws -quiet -progressive
# Do MCMC stuff
rhohat = (draw from posterior)
# now save the draws
sprintf fname "rho_%d.gdt", nbreaks
store "@fname" rhohat
endloop
endloop
</hansl>
This worked fine for me when I wrote it. However currently with the latest Windows
snapshot, after the first time through the loop, gretl stores successive data sets under
the name "(a)fname.gdt". The string is being updated properly, but not parsed by
the store command. If I insert the line
blah = "@fname"
between the sprintf and store commands, everything works fine.
Best,
Peter
[High Point University logo]
Dr. Peter M. Summers |
One University Parkway, High Point, NC 27268
Office: 336-841-9650 | Department of Economics
Choose to be extraordinary!(r)
[Facebook logo]<https://www.facebook.com/HighPointU>[Twitter
logo]<https://twitter.com/HighPointU>[Instagram
logo]<http://instagram.com/highpointu>[LinkedIn
logo]<https://www.linkedin.com/school/28206>
5:24 p.m.
On Sun, 17 Dec 2017, Summers, Peter wrote:
Hi all,
I was revisiting some code I wrote a few years ago, and came
across what is at least changed (and unexpected) behavior. I have
a script that loops over various numbers of possible structural
breaks, and does some MCMC sampling for each value. I want to
write the mcmc draws to separate data sets labeled by the number
of breaks.
Here's what I originally had:
<hansl>
loop ii=1..breaks --quiet
Nbreaks = ii-1
# do a bunch of stuff
loop draws = 1..ndraws -quiet -progressive
# Do MCMC stuff
rhohat = (draw from posterior)
# now save the draws
sprintf fname "rho_%d.gdt", nbreaks
store "@fname" rhohat
endloop
endloop
</hansl>
This worked fine for me when I wrote it. However currently with
the latest Windows snapshot, after the first time through the
loop, gretl stores successive data sets under the name
"(a)fname.gdt". The string is being updated properly, but not parsed
by the store command. If I insert the line
blah = "@fname"
between the sprintf and store commands, everything works fine.
I tried constructing the following test case (on the assumption that
the undefined identifier "nbreaks" was supposed to be the same as
the defined "Nbreaks":
<hansl>
set verbose off
nulldata 8
scalar breaks = 2
scalar ndraws = 3
loop ii=1..breaks --quiet
Nbreaks = ii-1
# do a bunch of stuff
loop draws = 1..ndraws --quiet --progressive
# Do MCMC stuff
scalar rhohat = draws
# now save the draws
sprintf fname "rho_%d.gdt", Nbreaks
store @fname rhohat
endloop
endloop
</hansl>
and I see what (I think) you mean!
If you use the function-form of sprintf; that is
fname = sprintf("rho_%d.gdt", Nbreaks)
in place of
sprintf fname "rho_%d.gdt", Nbreaks
It works OK; otherwise I see the error that you describe.
So, yes, this is a bug (associated with an old usage of "sprintf",
but a usage that supposedly still ought to work). I'll see what can
be done, but also I guess we should officially deprecate the
command-form of "sprintf" -- if we don't already do so.
Allin
6:01 p.m.
On Dec 17, 2017, at 5:24 PM, Allin Cottrell <cottrell(a)wfu.edu>
wrote:
> This worked fine for me when I wrote it. However currently with the latest Windows
snapshot, after the first time through the loop, gretl stores successive data sets under
the name "(a)fname.gdt". The string is being updated properly, but not parsed by
the store command. If I insert the line
>
> blah = "@fname"
>
> between the sprintf and store commands, everything works fine.
I tried constructing the following test case (on the assumption that the undefined
identifier "nbreaks" was supposed to be the same as the defined
"Nbreaks":
Yes, that was my email "correcting" hansl
If you use the function-form of sprintf; that is
fname = sprintf("rho_%d.gdt", Nbreaks)
in place of
sprintf fname "rho_%d.gdt", Nbreaks
It works OK; otherwise I see the error that you describe.
So, yes, this is a bug (associated with an old usage of "sprintf", but a usage
that supposedly still ought to work). I'll see what can be done, but also I guess we
should officially deprecate the command-form of "sprintf" -- if we don't
already do so.
My original code was actually (iirc) my first successful use of sprintf for this sort of
thing! I'd forgotten about its functional form, although I also recall you helping me
out with that too.
Thanks, Allin!
PS
6:14 p.m.
Am 17.12.2017 um 23:24 schrieb Allin Cottrell:
So, yes, this is a bug (associated with an old usage of
"sprintf", but a
usage that supposedly still ought to work). I'll see what can be done,
but also I guess we should officially deprecate the command-form of
"sprintf"
Could you run one of your shell-magic scripts to check which of the
function packages use that form? I'm a bit afraid of yet another
necessary round of updates, like we have had for turning boolean "="
into "==".
thanks,
sven
6:53 p.m.
On Mon, 18 Dec 2017, Sven Schreiber wrote:
Am 17.12.2017 um 23:24 schrieb Allin Cottrell:
> So, yes, this is a bug (associated with an old usage of
> "sprintf", but a usage that supposedly still ought to work). I'll
> see what can be done, but also I guess we should officially
> deprecate the command-form of "sprintf"
Could you run one of your shell-magic scripts to check which of
the function packages use that form? I'm a bit afraid of yet
another necessary round of updates, like we have had for turning
boolean "=" into "==".
Point taken, but this is a bit different. Using "=" as a test for
equality almost always worked OK, despite being deprecated. On the
other hand, the command-form of "sprintf" has not yet been
deprecated (so far as I can tell) yet in the context of a loop as in
Peter's script it has stopped working (properly).
So I'd say the first priority is to get the old form of sprintf
working again, if possible, while also officially deprecating it.
The second priority is then to future-proof function packages by
ensuring that they use the currently recommended form -- and in that
context, I agree, some "shell magic" would be helpful.
Allin
3:17 a.m.
On Sun, 17 Dec 2017, Allin Cottrell wrote:
The second priority is then to future-proof function packages by
ensuring that they use the currently recommended form -- and in that
context, I agree, some "shell magic" would be helpful.
A proper implementation would have to use our internal tokenizer, but for
a rough check this bash fragment may come in useful (it only catches "if"s
and you do get quite a lot of false positives, but hey)
<bash>
function chk {
FILE=$1
cat -n $FILE | grep "if " | grep '=' | grep -v -e
"[!<>=]=" > /tmp/xxx
if [ -s /tmp/xxx ]; then
echo "--- $FILE"
sed /tmp/xxx -e 's/^/\t/'
# else
# echo "--- $FILE ok"
fi
}
for f in $(find . -name "*.inp") ; do chk "$f"; done
</bash>
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
3:52 a.m.
Am 18.12.2017 um 09:17 schrieb Riccardo (Jack) Lucchetti:
On Sun, 17 Dec 2017, Allin Cottrell wrote:
> The second priority is then to future-proof function packages by
> ensuring that they use the currently recommended form -- and in that
> context, I agree, some "shell magic" would be helpful.
A proper implementation would have to use our internal tokenizer, but
for a rough check this bash fragment may come in useful (it only catches
"if"s a
Ahem, could it be you're talking about the == thing here? The current
(or future) problem was the 'sprintf' command.
What I'm saying is that the number of function packages that use the
command form of sprintf might have a bearing on the decision whether (or
when) to deprecate it. The transition from = to == was actually very
slow and relatively painful -- I'm not even sure it's finished already.
I wouldn't like to repeat that experience if it's not really necessary.
And/or several deprecations should be lumped together I guess.
cheers,
sven
5:35 a.m.
On Mon, 18 Dec 2017, Sven Schreiber wrote:
Am 18.12.2017 um 09:17 schrieb Riccardo (Jack) Lucchetti:
> On Sun, 17 Dec 2017, Allin Cottrell wrote:
>
>> The second priority is then to future-proof function packages by ensuring
>> that they use the currently recommended form -- and in that context, I
>> agree, some "shell magic" would be helpful.
>
> A proper implementation would have to use our internal tokenizer, but for a
> rough check this bash fragment may come in useful (it only catches "if"s a
Ahem, could it be you're talking about the == thing here? The current (or
future) problem was the 'sprintf' command.
Ah, ok, sorry.
I think the following could be a base for automatic sanitizing of sprintf
<bash>
# converts sprintf commands into sprintf functions
sed $1 -e 's/sprintf \([A-Za-z_0-9]*\) \(.*\)/string \1 = sprintf(\2)/'
</bash>
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
6:50 a.m.
On Mon, 18 Dec 2017, Riccardo (Jack) Lucchetti wrote:
On Mon, 18 Dec 2017, Sven Schreiber wrote:
> Am 18.12.2017 um 09:17 schrieb Riccardo (Jack) Lucchetti:
>> On Sun, 17 Dec 2017, Allin Cottrell wrote:
>>
>>> The second priority is then to future-proof function packages by ensuring
>>> that they use the currently recommended form -- and in that context, I
>>> agree, some "shell magic" would be helpful.
>>
>> A proper implementation would have to use our internal tokenizer, but for
>> a rough check this bash fragment may come in useful (it only catches
"if"s
>> a
>
> Ahem, could it be you're talking about the == thing here? The current (or
> future) problem was the 'sprintf' command.
Ah, ok, sorry.
I think the following could be a base for automatic sanitizing of sprintf
<bash>
# converts sprintf commands into sprintf functions
sed $1 -e 's/sprintf \([A-Za-z_0-9]*\) \(.*\)/string \1 = sprintf(\2)/'
</bash>
better (preserves indentation and adds a comment)
<sprintf.sed>
s/^\([ \t]*\)sprintf \([A-Za-z_0-9]*\) \(.*\)/\1# was: sprintf \2 \3\n\1string \2 =
sprintf(\3)\n/
</sprintf.sed>
<bash>
sed $1 -f sprintf.sed
</bash>
so for example
<hansl>
sprintf ii "%d", i-2
</hansl>
becomes
<hansl>
# was: sprintf ii "%d", i-2
string ii = sprintf("%d", i-2)
</nansl>
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
2:01 p.m.
Am 18.12.2017 um 12:50 schrieb Riccardo (Jack) Lucchetti:
On Mon, 18 Dec 2017, Riccardo (Jack) Lucchetti wrote:
> I think the following could be a base for automatic sanitizing of
sprintf
better (preserves indentation and adds a comment)
<sprintf.sed>
s/^\([ \t]*\)sprintf \([A-Za-z_0-9]*\) \(.*\)/\1# was: sprintf \2
\3\n\1string \2 = sprintf(\3)\n/
</sprintf.sed>
<bash>
sed $1 -f sprintf.sed
</bash>
so for example
<hansl>
sprintf ii "%d", i-2
</hansl>
becomes
<hansl>
# was: sprintf ii "%d", i-2
string ii = sprintf("%d", i-2)
</nansl>
OK, very nice -- so how many packages are affected?
thanks,
sven
3:07 p.m.
On Mon, 18 Dec 2017, Sven Schreiber wrote:
Am 18.12.2017 um 12:50 schrieb Riccardo (Jack) Lucchetti:
> On Mon, 18 Dec 2017, Riccardo (Jack) Lucchetti wrote:
>> I think the following could be a base for automatic sanitizing of sprintf
> better (preserves indentation and adds a comment)
>
> <sprintf.sed>
> s/^\([ \t]*\)sprintf \([A-Za-z_0-9]*\) \(.*\)/\1# was: sprintf \2
> \3\n\1string \2 = sprintf(\3)\n/
> </sprintf.sed>
>
> <bash>
> sed $1 -f sprintf.sed
> </bash>
>
> so for example
>
> <hansl>
> sprintf ii "%d", i-2
> </hansl>
>
> becomes
>
> <hansl>
> # was: sprintf ii "%d", i-2
> string ii = sprintf("%d", i-2)
> </nansl>
OK, very nice -- so how many packages are affected?
I have no idea :D
I could conjure up a script for (a) downloading a package (b) extracting
the hansl code (c) sanitizing it and (d) repackage, but it'd be quite a
job.
Allin, any idea on automating the process somehow?
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
4:16 p.m.
On Mon, 18 Dec 2017, Sven Schreiber wrote:
Am 18.12.2017 um 12:50 schrieb Riccardo (Jack) Lucchetti:
> On Mon, 18 Dec 2017, Riccardo (Jack) Lucchetti wrote:
>> I think the following could be a base for automatic sanitizing of sprintf
> better (preserves indentation and adds a comment)
>
> <sprintf.sed>
> s/^\([ \t]*\)sprintf \([A-Za-z_0-9]*\) \(.*\)/\1# was: sprintf \2
> \3\n\1string \2 = sprintf(\3)\n/
> </sprintf.sed>
>
> <bash>
> sed $1 -f sprintf.sed
> </bash>
>
> so for example
>
> <hansl>
> sprintf ii "%d", i-2
> </hansl>
>
> becomes
>
> <hansl>
> # was: sprintf ii "%d", i-2
> string ii = sprintf("%d", i-2)
> </nansl>
OK, very nice -- so how many packages are affected?
I agree, very nice. However, I don't think we need to get too worked
up about the command-form of sprintf. Our internal "tokenizer" can
quite happily convert it to the function form -- that's now in git.
Moreover, on closer inspection I don't think it was the sprintf
command that was responsible for the bad behavior in the script that
Peter posted recently; rather it was a bug in the "progressive" loop
code, which is now fixed.
(By the way, the --progressive option to loops dates back to before
gretl had matrices. Now that hansl is a lot more functional I'd be
happy to get rid of --progressive; everything it does can be done
quite easily using an ordinary loop plus matrices. But that would be
seriously backward-incompatible so I won't push it.)
Anyway, I've now purged the command-form of sprintf from the
documentation, replacing the command-reference page with a pointer
to the function-form. I think that's enough for now. We can issue a
deprecation without requiring that it be changed any time soon in
affected function packages: "shouldn't be used in newly written
hansl code" would be sufficient so far as I'm concerned.
The "=" versus "==" business was more pressing since the resulting
ambiguity gave rise to a pretty much insoluble parsing problem in
certain special cases. That's not an issue with sprintf command
versus sprintf function.
Allin
2485
days inactive
2486
days old
11 comments
4 participants
participants (4)
-
Allin Cottrell -
Riccardo (Jack) Lucchetti -
Summers, Peter -
Sven Schreiber