10:12 p.m.
I said I'd have gretl 1.8.4 out by now, but I found that fixing
problems with the user-function GUI led me further "down the
rabbit hole" than I expected.
I discovered quite a lot of code in lib/src/gretl_func.c, and also
in the associated GUI C files, that was partially duplicated
and/or not readily comprehensible, which may well count for the
"flakiness" that Sven mentioned.
I've therefore done a good deal of clean-up (plus addition of
comments) in those areas. I've also relaxed what I think was the
most awkward limitation in the function-package design up to gretl
1.8.3, namely the fact that a package could only have one public
interface. Now packages can contain as many public (and private)
functions as you like.
This is all good, but as with all new code it needs some testing.
So if anyone can find some time over the next few days to beat on
anything to do with editing or calling function packages, I'd be
very grateful. Then we'll get gretl 1.8.4 out.
Reminder: cleaned-up function packages for use with current CVS
gretl can be found at
http://www.wfu.edu/~cottrell/newgfn/
Two further comments:
* At present you define the list of functions belonging to a
package at the time of initial construction of the package; and
there's no way, in the GUI, to add or remove functions from a
package. I plan to remedy this before long, but probably not for
the 1.8.4 release.
* Now that packages can offer more than one public function
there's a potential ambiguity over the "Help text". I've resolved
this for now by locating the Help at the package level. That is,
suppose you have a package "sfa" that has public functions
"sfa_mod" and "sfa_hetmod" (I'm taking a possible reorganization
of Gordon Hughes's contribution as an example). In that case the
help text that you enter in the GUI package editor should cover
both functions. If the user does "help sfa_mod" or "help
sfa_hetmod", in either case they'll get a redirect to the help
text for the parent package.
Allin.
9:06 a.m.
Allin Cottrell schrieb:
I said I'd have gretl 1.8.4 out by now, but I found that fixing
problems with the user-function GUI led me further "down the
rabbit hole" than I expected.
I discovered quite a lot of code in lib/src/gretl_func.c, and also
in the associated GUI C files, that was partially duplicated
and/or not readily comprehensible, which may well count for the
"flakiness" that Sven mentioned.
let's hope for the best!
I've therefore done a good deal of clean-up (plus addition of
comments) in those areas. I've also relaxed what I think was the
most awkward limitation in the function-package design up to gretl
1.8.3, namely the fact that a package could only have one public
interface. Now packages can contain as many public (and private)
functions as you like.
what about the package name then? It used to be that the file name had
to match the name of the public function. (Which I always found mildly
annoying, but could live with.) what's the situation now?
This is all good, but as with all new code it needs some testing.
So if anyone can find some time over the next few days to beat on
anything to do with editing or calling function packages, I'd be
very grateful. Then we'll get gretl 1.8.4 out.
I think I will have time to test some of these things until tomorrow.
...
* At present you define the list of functions belonging to a
package at the time of initial construction of the package; and
there's no way, in the GUI, to add or remove functions from a
package. I plan to remedy this before long, but probably not for
the 1.8.4 release.
Actually I dont think this is a big problem. You can always export an
existing package to a flat script and then start a new package by adding
or removing some components. So personally I wouldn't give this feature
high priority.
I think this is also related to the problem of versioning of the
packages on the public server: suppose there's a package written and
uploaded by author A, and used in productive work by user U. Then author
A develops the package further and uploads the new version to the
server, superseding the old one. (Or is the old version kept on the
server parallel to the new one, I don't know right now?) And user U
suddenly is presented with new interfaces, different APIs and possibly
new bugs etc. etc. I think this problem has to be taken into account if
you want to promote the use of packages.
thanks,
sven
10:22 a.m.
On Wed, 26 Aug 2009, Sven Schreiber wrote:
Allin Cottrell schrieb:
> I said I'd have gretl 1.8.4 out by now, but I found that fixing
> problems with the user-function GUI led me further "down the
> rabbit hole" than I expected.
>
> I discovered quite a lot of code in lib/src/gretl_func.c, and also
> in the associated GUI C files, that was partially duplicated
> and/or not readily comprehensible, which may well count for the
> "flakiness" that Sven mentioned.
let's hope for the best!
;-)
> I've therefore done a good deal of clean-up (plus addition
of
> comments) in those areas. I've also relaxed what I think was the
> most awkward limitation in the function-package design up to gretl
> 1.8.3, namely the fact that a package could only have one public
> interface. Now packages can contain as many public (and private)
> functions as you like.
what about the package name then? It used to be that the file name had
to match the name of the public function. (Which I always found mildly
annoying, but could live with.) what's the situation now?
That limitation is gone. Here's the current situation: for a new
package, the package name is derived from the filename, which must
meet certain criteria. The basename must start with a letter,
contain nothing but ASCII letters, numerals (preferably not), and
the underscore character; and it must be less than 32 characters
in length. It also must have suffix ".gfn".
> * At present you define the list of functions belonging to a
> package at the time of initial construction of the package; and
> there's no way, in the GUI, to add or remove functions from a
> package. I plan to remedy this before long, but probably not for
> the 1.8.4 release.
Actually I dont think this is a big problem. You can always export an
existing package to a flat script and then start a new package by adding
or removing some components. So personally I wouldn't give this feature
high priority.
I agree, you can do that. But since we have a GUI editor it might
as well be properly functional. Adding or deleting interfaces is
something you may want to do when working on a complex package and
it's a bit of a pain to have to shuttle back and forth between
package-form and script-form.
I think this is also related to the problem of versioning of the
packages on the public server: suppose there's a package written
and uploaded by author A, and used in productive work by user U.
Then author A develops the package further and uploads the new
version to the server, superseding the old one. (Or is the old
version kept on the server parallel to the new one, I don't know
right now?) And user U suddenly is presented with new
interfaces, different APIs and possibly new bugs etc. etc. I
think this problem has to be taken into account if you want to
promote the use of packages.
Yes, I agree this is something we'll have to think about.
Allin.
8:07 p.m.
New subject: Bug in Exponential Smoothing
Hello Devs,
I believe there is a bug in the Exponential Smoothing code. Using the
example from Makridakis, Wheelwright, Hyndman, "Forecasting Methods and
Applications," 3e, table 4-3 on page 151, we should get,
When running the same in GRETL using
for each series, we get
When displayed we get,
To get the same MSE as Makridakis we need,
? printf "MSE for SES1 is: %9.3f\n", sum((ship-ses1(-1))^2)/10
MSE for SES1 is: 3438.332
? printf "MSE for SES2 is: %9.3f\n", sum((ship-ses2(-1))^2)/10
MSE for SES2 is: 4347.237
? printf "MSE for SES3 is: %9.3f\n", sum((ship-ses3(-1))^2)/10
MSE for SES3 is: 5039.368
The calculated EMAs are correct, but need to be lagged.
Which provides a graph that matches their figure 4-7
--Peter
9:25 p.m.
New subject: Bug in Exponential Smoothing
On Sun, 13 Sep 2009, peter wrote:
I believe there is a bug in the Exponential Smoothing code. Using
the
example from Makridakis, Wheelwright, Hyndman, "Forecasting Methods and
Applications," 3e, table 4-3 on page 151, ...
[ gretl produces results which have to be lagged one period to
match those in the textbook. ]
Yes, you're right. The exact specification of the EMA formula
seems kind of arbitrary to me, but given that we offer EMA at all
we should abide by the most common convention.
I think this should now be fixed in CVS.
Allin Cottrell
10:06 p.m.
New subject: Bug in Exponential Smoothing
Allin Cottrell wrote:
On Sun, 13 Sep 2009, peter wrote:
> I believe there is a bug in the Exponential Smoothing code. Using the
> example from Makridakis, Wheelwright, Hyndman, "Forecasting Methods and
> Applications," 3e, table 4-3 on page 151, ...
>
[ gretl produces results which have to be lagged one period to
match those in the textbook. ]
Yes, you're right. The exact specification of the EMA formula
seems kind of arbitrary to me,
Agreed.
but given that we offer EMA at all
we should abide by the most common convention.
I think this should now be fixed in CVS.
Thanks very much.
Peter
11:18 a.m.
Sven Schreiber schrieb:
> I've therefore done a good deal of clean-up (plus addition of
> comments) in those areas. I've also relaxed what I think was the
> most awkward limitation in the function-package design up to gretl
> 1.8.3, namely the fact that a package could only have one public
> interface. Now packages can contain as many public (and private)
> functions as you like.
what about the package name then? It used to be that the file name had
to match the name of the public function. (Which I always found mildly
annoying, but could live with.) what's the situation now?
I noticed that the name can be freely chosen now, apparently.
> This is all good, but as with all new code it needs some testing.
> So if anyone can find some time over the next few days to beat on
> anything to do with editing or calling function packages, I'd be
> very grateful. Then we'll get gretl 1.8.4 out.
I think I will have time to test some of these things until tomorrow.
First round of test:
Everything worked as expected during (the first) packaging.
Running the function package produced the error "needs gretl 1.8.4" (I'm
running today's cvs); so I wanted to edit the package and tried lowering
the requirement to 1.8.3 (via the clickable number field" in the GUI).
But when I try to save the changed version, again I get the error
"function needs gretl 1.8.4". This looks like a bug to me.
(BTW, when I click "close" instead of "save", gretl asks "want to
save
the changes?", I click "yes" and gretl gives me the same error dialog,
but when I dismiss this error dialog, the package is closed
nevertheless, so I presume that any other changes that I might have made
would have been lost.)
So I start a new package to incorporate the lowered requirement. But --
gretl complains "no functions available for packaging". Ok, I did delete
the previous package (that had just failed) from my machine via gretl's
package management GUI, but the functions that I had run earlier in
order to make them available for packaging should still be available, no?
So I run the script with the functions again as a prelude to packaging.
This runs without error (well it just contains the functions, no actual
function calls). Then again I want to create a new package, but still
the same error, "no functions available".
In general my impression with the last couple of gretl versions has been
that errors of almost any kind seem to corrupt the function "namespace"
somehow. This does not seem to have been solved completely even now.
Don't get me wrong, I don't think it's a major issue, because the
packaging of functions is not a frequent task and so re-starting gretl
for that is bearable.
Ok, time to re-start gretl. Then running the script with the functions
and packaging works fine (and I remembered to change the requirement to
1.8.3). Saving & starting the package in the CWD works fine via GUI. The
expected results (printed output including ascii graphs and result
matrix) are produced ok.
However -- I open this function package again for editing, and add two
words to the help text. When I click "save" an error window pops up:
"data error". (BTW, all in German re-translated on the spot, hopefully
recognizable.) The same happens when I try to edit the package
description. And then --without deleting any package in between or
anything else-- when I try to create a new package I get the old message
"no functions available."
The "data error" thing also happens when I re-start gretl and then want
to edit a package. (Actually it seems that the old help text was deleted
by gretl after I tried extending it, because the field is empty now
after re-starting gretl and opening the package. Although it appeared
that gretl didn't want to allow an empty help text when I first created
the package.)
Bottom line: editing of existing packages currently still seems
impossible. But when the package is created ok it works fine.
I wonder whether it may be too ambitious to have this GUI thing for
packaging (as opposed to have a GUI for running the package, that's a
great feature). Package authors are probably scripting-savvy enough so
that they could be expected to prepare one special script with a special
package mark-up (like <helper-function-start> ... <helper-function-end>
or whatever, you get the picture), and when this marked-up script is run
a package is produced. Would that help in eradicating the bugs?
thanks,
sven
2:27 p.m.
Sven Schreiber schrieb:
(that's a nicely euphonious phrase ;-)
First round of test:
Everything worked as expected during (the first) packaging.
Running the function package produced the error "needs gretl 1.8.4" (I'm
running today's cvs); so I wanted to edit the package and tried lowering
the requirement to 1.8.3 (via the clickable number field" in the GUI).
But when I try to save the changed version, again I get the error
"function needs gretl 1.8.4". This looks like a bug to me.
In CVS I've now shifted the default minimum version to 1.8.3; I'll
put it back to 1.8.4 at release time.
As for this bug, and the further ones you reported: sorry, I
should have done some testing myself before I asked anyone else to
do so, but thanks for blazing the trail!
Although the buggy effects were various they almost all stemmed
from a single, simple source, now fixed in CVS and snapshot. A
new piece of code that just wasn't ready for prime time.
So I start a new package to incorporate the lowered requirement.
But -- gretl complains "no functions available for packaging".
Ok, I did delete the previous package (that had just failed)
from my machine via gretl's package management GUI, but the
functions that I had run earlier in order to make them available
for packaging should still be available, no?
This one is a separate (and long-standing) issue, namely what
exactly should happen when you delete a function package via the
GUI? At present all that happens is that the function _file_ is
deleted, but it makes sense that the package should also be
unloaded from memory. Then what should happen to the functions
that the package comprised? Either they are unloaded too, or
(and this is the behavior you seem to regard as most intuitive)
they are retained in memory but disconnected from the
scrapped package (so they're available for packaging again).
That should be asy to arrange.
I wonder whether it may be too ambitious to have this GUI thing
for packaging (as opposed to have a GUI for running the package,
that's a great feature). Package authors are probably
scripting-savvy enough so that they could be expected to prepare
one special script with a special package mark-up (like
<helper-function-start> ... <helper-function-end> or whatever,
you get the picture), and when this marked-up script is run a
package is produced. Would that help in eradicating the bugs?
Possibly, though I'd like to persevere a little longer. I'm not
sure I have all the bugs out of the GUI editor yet, but with
current CVS things should look a lot better!
Allin.
3:30 p.m.
On Wed, 26 Aug 2009, Allin Cottrell wrote:
[What] exactly should happen when you delete a function package
via the GUI? At present all that happens is that the function
_file_ is deleted, but it makes sense that the package should
also be unloaded from memory. Then what should happen to the
functions that the package comprised? Either they are unloaded
too, or ... they are retained in memory but disconnected from
the scrapped package (so they're available for packaging again).
The second option is now implemented in CVS. Deleting a package in
the GUI frees its functions for other uses.
Allin.
5:38 a.m.
Allin Cottrell schrieb:
On Wed, 26 Aug 2009, Allin Cottrell wrote:
> [What] exactly should happen when you delete a function package
> via the GUI? At present all that happens is that the function
> _file_ is deleted, but it makes sense that the package should
> also be unloaded from memory. Then what should happen to the
> functions that the package comprised? Either they are unloaded
> too, or ... they are retained in memory but disconnected from
> the scrapped package (so they're available for packaging again).
The second option is now implemented in CVS. Deleting a package in
the GUI frees its functions for other uses.
I think that's a good choice -- I will do some more testing soon.
-sven
7:31 a.m.
Sven Schreiber schrieb:
I think that's a good choice -- I will do some more testing
soon.
the old bugs are gone AFAICT, great work!
Then however I did something mean:
After saving the package (and having run it successfully) I created
another function with the same name as the public interface of the
package. Ran the new function (it doesn't do anything, basically just a
dummy). Then ran the previously saved function again: The result is not
the one specified by the function within the package, but by the new
version. So the result is rubbish.
As I said, this is mean-spirited and I guess it's not new behavior.
Ideally, the package should be encapsulated from other scripts that an
unsuspecting user creates, but I don't think this is a blocker for a new
release.
another minor and unrelated issue:
An integer argument with a range such as "int freqpoints[0:500:100]"
gives in the GUI an entry field with clickable up-down arrows, which is
good.
But an argument w/o range such as "int lagorder[3]" gives no clickable
arrows (a drop-down-list-style arrow is there but greyed out), though I
cannot see why they couldn't be there as well.
(For non-integer arguments it's only a bit strange that there's also a
greyed-out arrow, but that's only cosmetic.)
thanks,
sven
8:28 p.m.
On Wed, 26 Aug 2009, Sven Schreiber wrote:
I wonder whether it may be too ambitious to have this GUI thing
for packaging (as opposed to have a GUI for running the package,
that's a great feature). Package authors are probably
scripting-savvy enough so that they could be expected to prepare
one special script with a special package mark-up (like
<helper-function-start> ... <helper-function-end> or whatever,
you get the picture), and when this marked-up script is run a
package is produced. Would that help in eradicating the bugs?
I'd like to try to answer this in a bit more depth. Particularly
since, if Jack were with us (soon may he return!), I'm sure he'd
agree with you in general terms.
My first point pertains to your final question: "Would that
[bypassing a GUI function packager] help in eradicating the bugs?"
The answer to that is No. The bugs you found were not located at
the GUI level, they were at the level of the underlying code in
gretl_func.c that handles the relationships between individual
user-defined functions and packages.
I think I now have that code in much better shape, but there were
some easily-fixed, surface-level bugs in my new additions.
Anyway, coming back to your main point: is there really any need
for, or point to, the GUI packager?
Well, let's take this from the ground up. A gretl function is
something you might write on an ad hoc basis for use in a complex
script. In general there's no need to attach meta-information
such as author, date, version, description, help text, example
script, etc. But I take for granted that we _do_ need that sort
of information in a package. So we need some structured means of
recording meta-information.
Now, the method I originally chose for recording that sort of
information is XML, using a specially designed Document Type
Definition (DTD), namely gretlfunc.dtd (it's in the source
package). A current gretl function package (.gfn file) is an XML
document conforming to that DTD which contains one or more
function definitions. And the GUI packager is supposed to let you
produce such files fairly painlessly.
We can debate whether XML is the best way to go. It has the
drawback that some characters commonly found in gretl scripts have
to be escaped, which makes it awkward to hand-edit a script inside
a .gfn file unless you know a little about XML. But it has the
same sort of advantage as wearing a seat-belt: we get some useful
automatic protection by validating a putative function package
against the DTD.
Yes, this only concerns the gross syntatical features of the
package: a .gfn file could validate against the DTD yet be total
rubbish semantically; we need some intelligent human intervention
to validate packages in a substantive sense.
All the same, the functionality offered by libxml is useful. If
we weren't using XML we'd have to write custom, ad hoc, routines
to parse package files containing custom, ad hoc, representation
of the structural elements of a package. As a coder, I like the
XML seat-belt!
Plus, we have some history: for some time now gretl has supported
XML package files (however imperfectly -- and especially
imperfectly in the 1.8.3 release, alas!). The ideal of backward
compatibility says we should continue to support such files.
Moving forward, should we in addition support a quite different
form of package file, or should we try to augment the .gfn format
so that it can do more of what we want?
I'm trying to explore the latter option. I don't want to get into
embedding PDF documentation within an XML file -- Microsoft-style
yuck! -- but I can envisage a .gfn file that has an external
reference to PDF documentation, with the .gfn and the PDF wrapped
in a zip file. That way, I hope to keep some continuity while
supporting new features.
Allin.
5:50 a.m.
Allin Cottrell schrieb:
On Wed, 26 Aug 2009, Sven Schreiber wrote:
My first point pertains to your final question: "Would that
[bypassing a GUI function packager] help in eradicating the bugs?"
The answer to that is No.
Then I think there's no point in scrapping the GUI.
...
I'm trying to explore the latter option. I don't want to get
into
embedding PDF documentation within an XML file -- Microsoft-style
yuck! -- but I can envisage a .gfn file that has an external
reference to PDF documentation, with the .gfn and the PDF wrapped
in a zip file. That way, I hope to keep some continuity while
supporting new features.
I don't really think having embedded full-fledged documentation is
necessary. For example, the gretl pdf user guide is very important, but
it's perfectly fine that it's a separate file downloadable from somewhere.
That was my idea when I suggested using a wiki for the package
documentation on the users list (although I think I was a little
misunderstood by others). There would be a link from the gretl main page
to the package wiki, and package authors could upload documentation
about their packages to the wiki, complementing the embedded help texts
and sample scripts within the packages.
Although jack hasn't answered yet, I guess that such a package wiki
could be hosted on the general gretl wiki that he set up.
cheers,
sven
5543
days inactive
5562
days old
12 comments
3 participants
participants (3)
-
Allin Cottrell -
peter -
Sven Schreiber