8:44 p.m.
On Thu, 14 Jul 2005, Itzak Ademic wrote:
I've just discovered GReTL.
Before I plunge into the code,
I wonder if there is a FAQ, plan or just an attitude to shadow, or avoid,
existing products like TSP, FAME, RATS, TROLL, SAS, LISREL, Shazam...
Why not contribute to R or Octave?
Is there a wishlist for capabilities, broader directions, buglist?
Very good questions.
Up till fairly recently I've been "following my nose" in gretl
development, responding to user requests and to things I have found
interesting. This spring I went and visited gretl enthusisasts in
Italy, Spain and Poland, and I have a bit more direction. The main
short- to medium-term requests I heard related to better support for
forecasting, seasonal ARMA models, better support for VARs, and (a
bit more speculatively) a mechanism for manipulation of user-defined
vectors/matrices so people can roll their own estimators and tests
(perhaps along the lines of Doornik's Ox).
As for forecasting, that's now pretty much done in the latest
release. In gretl CVS we have seasonal ARMA and considerably
enhanced support for VARs. We don't yet have a serious start on
user-defined matrix manipulation, though many of the required basic
functions are there in gretl_matrix.c.
More generally, the question is, what level of sophistication is
gretl aiming for? My original idea (as I recall) was that I wanted
gretl to be an excellent tool for undergraduate-level econometrics.
I think we've pretty much got there, but the target has moved.
I think the notion is: If gretl provides a good interface for doing
econometric work, why stop at the basics? The value of gretl, even
to undergrads, will be enhanced if they can continue to use this
program for professional work.
"Shadowing or avoiding commercial products"
What I have here is more of an attitude than a definite plan, but
I'd like for gretl to be able to do much of what Eviews can, but
better(!). I have great respect for RATS. I have used RATS (under
dosemu) to test the output of some gretl routines. I don't think
gretl (at least in the medium term) can do all that RATS does, but I
do think we can have a greatly superior user interface.
"Why not contribute to R or Octave?"
Fair question. Again, I have great respect for R, but it has a hell
of a learning curve; I really blanch at the idea of asking my
students to work with R. And personally, I find I always have to
look everything up in the docs before I can work with data in R
(though I have done my homework from time to time, and have used R
output as a benchmark against which to test gretl for some
functionality). So there's ease of use. But also, gretl is ahead
of R in some areas of econometric functionality. Last time I
checked, R did not support FIML estimation of systems of equations.
Octave is a good program, but it's not particularly oriented towards
econometrics. I would happily "steal" Ocatve's matrix manipulation
code, except that Octave is written in C++ while gretl is in C
(though this need not be absolute barrier).
I'll try to summarize: My (immodest) notion is that gretl should be
an extremely user-friendly econometrics (and to some extent, general
statistics) package with true cross-platform GUI functionality and
also solid support for scripting. It should be a flagship for
free, open-source software in the statistical domain -- a program
which many users might prefer to expensive proprietary alternatives
(I have some evidence that this is true already).
I'm cc'ing this to the gretl users mailing list
http://ricardo.ecn.wfu.edu/mailman/listinfo/gretl-users
If you're sufficiently interested, I'd urge you to continue the
correspondence there. One of the things I learned this spring is
that at this point gretl is becoming a community project, and it
would be good to have others put in their views.
Allin Cottrell
6:45 a.m.
New subject: gretl development roadmap
El Viernes, 15 de Julio de 2005 02:44, Allin Cottrell escribió:
...
As for forecasting, that's now pretty much done in the latest
release. In gretl CVS we have seasonal ARMA and considerably
enhanced support for VARs. We don't yet have a serious start on
user-defined matrix manipulation, though many of the required basic
functions are there in gretl_matrix.c.
More generally, the question is, what level of sophistication is
gretl aiming for? My original idea (as I recall) was that I wanted
gretl to be an excellent tool for undergraduate-level econometrics.
In 2001 I was several months looking for an econometrics package I could offer
to my undergraduate students for free. I was just discovering "open source"
and Linux, and I was really surprised when I found Gretl. I liked the
structure of gretl just when I saw it, mainly because of its so user-friendly
interface. The Windows version of gretl was full of bugs, so I began to
collaborate sending very long reports of bugs. Now I think we have reached an
stage in which gretl is very stable and very appropiate for teaching
Econometrics in an undergraduate level. It covers now almost all the
undergraduate programs we teach at our University. (With the seasonal ARMA,
it will also cover the programs of the "Time series" subjects, thank you
Allin).
I think we've pretty much got there, but the target has moved.
I think the notion is: If gretl provides a good interface for doing
econometric work, why stop at the basics? The value of gretl, even
to undergrads, will be enhanced if they can continue to use this
program for professional work.
I do not agree with Allin at this subject. Postgraduate econometrics work
usually requires programming at a very specific level. We have a lot of
computer "languages" for doing econometrics at this specific level now:
Gauss, Rats (including the compile mode), Ox, Splus, R, Octave, ...
I the 90's I was a Rats user, and when I discovered the "open source"
philosophy, I thought in porting some of my programs to an "open source"
language.
R is a language and environment for statistical computing and graphics (see
http://www.r-project.org ) which is, as gretl, "open source and free
software", has a very strong user community, a very structured development
team, and a quality control that ensures that the published packages
(libraries and functions) has a very low probability of errors.
I think we should not dedicate much effort in adding new features to the gretl
console, but it will be better for the user to get a better relation with the
R language. At present, we can open a dataset in gretl and through
"Utilities/Start Gnu R" we can open an R session and read our dataset. It is
very usefull, because now we can operate with these data with all the
commands and functions of R. But now we would need a way to recover some of
the R results (yhats, residuals, coefficients, ..) passing them again to
Gretl.
Having this type of link gretl<-->R working, we will avoid the necessity to
program all the features that are already programmed in R. For example,
suppose you need to use the Kalman filter, Structural Time Series Models, or
whatever matricial operation, all of these can be done in R (reading the
manuals, of course).
I think Gretl and R are now complementary programs. Gretl for the basics and R
for the advanced econometrics. Increasing the capabilities of the gretl
console (gretlcli) will lead also the user "to look everything up in the
docs before he can work with data".
We should avoid that the user have to learn two different "languages" for
doing the same things (in R and gretl), and so we should avoid that gretl and
R compete for the users.
I'll try to summarize: My (immodest) notion is that gretl should be
an extremely user-friendly econometrics (and to some extent, general
statistics) package with true cross-platform GUI functionality and
also solid support for scripting. It should be a flagship for
free, open-source software in the statistical domain -- a program
which many users might prefer to expensive proprietary alternatives
(I have some evidence that this is true already).
I agree with almost all above. I only suggest that we should not define
another scripting language very different from the existing ones (specially
from the R language).
--
Ignacio Díaz-Emparanza
Dpto. de Economía Aplicada III (Econometría y Estadística)
UPV-EHU
http://www.bl.ehu.es/~etpdihei/
4:18 a.m.
New subject: gretl development roadmap
On Fri, 15 Jul 2005, Ignacio Díaz-Emparanza wrote:
> I think we've pretty much got there, but the target has
moved.
> I think the notion is: If gretl provides a good interface for doing
> econometric work, why stop at the basics? The value of gretl, even
> to undergrads, will be enhanced if they can continue to use this
> program for professional work.
I do not agree with Allin at this subject. Postgraduate econometrics work
usually requires programming at a very specific level. We have a lot of
computer "languages" for doing econometrics at this specific level now:
Gauss, Rats (including the compile mode), Ox, Splus, R, Octave, ...
I the 90's I was a Rats user, and when I discovered the "open source"
philosophy, I thought in porting some of my programs to an "open source"
language.
R is a language and environment for statistical computing and graphics (see
http://www.r-project.org ) which is, as gretl, "open source and free
software", has a very strong user community, a very structured development
team, and a quality control that ensures that the published packages
(libraries and functions) has a very low probability of errors.
I think we should not dedicate much effort in adding new features to the gretl
console, but it will be better for the user to get a better relation with the
R language. At present, we can open a dataset in gretl and through
"Utilities/Start Gnu R" we can open an R session and read our dataset. It is
very usefull, because now we can operate with these data with all the
commands and functions of R. But now we would need a way to recover some of
the R results (yhats, residuals, coefficients, ..) passing them again to
Gretl.
Having this type of link gretl<-->R working, we will avoid the necessity to
program all the features that are already programmed in R. For example,
suppose you need to use the Kalman filter, Structural Time Series Models, or
whatever matricial operation, all of these can be done in R (reading the
manuals, of course).
I think Gretl and R are now complementary programs. Gretl for the basics and R
for the advanced econometrics. Increasing the capabilities of the gretl
console (gretlcli) will lead also the user "to look everything up in the
docs before he can work with data".
We should avoid that the user have to learn two different "languages" for
doing the same things (in R and gretl), and so we should avoid that gretl and
R compete for the users.
I have heard Ignacio's point many times, but I'm still not convinced. I
had nearly the same conversation with my friend Christine Choirat (of
OxJapi fame) a few months ago. She's an extremely good programmer, and her
line was basically "why re-invent the wheel?". Instead on working on
gretl, I remember she was planning to create a Tcl/Tk wrapper around R
which would mimic Pc-Give as much as possible.
Well, Allin already outlined a few reasons why the wheel is still worthy
of being worked on. First, R is hard. I consider myself above the average
econometrician when it comes to computer languages (perhaps immodestly),
and still working with R is a PITA for me. R's syntax is very
idiosyncratic, and either you work with it all the time, or you spend 1
hour doing your work and 9 looking at the manual. This is most likely a
big problem for someone who is not a full-time econometrician, but it is
likely to be a big hurdle for people who do applied research who want
something that "just works". Remember, these people form the vast majority
of the users of econometrics/statistical packages. Example: I had a
colleague who needed the HP filter. Fairly standard, no? It took me less
time to add it into gretl (with Allin's help) an give him the upgraded
package than it would have taken having him download and set up R,
learning how to import his data, perform the filtering, and going back to
the applications he normally uses. And once this is done, it's done for
everyone. Rather than the fish, you give people the fishing rod.
If the functionality of gretl as a R front-end can be enhanced, that would
be a very cool thing to have (for me at least), but there's no reason to
be tied to it for the less trivial computations. Besides, if you look at
the source, the infrastructure for adding more sophisticated stuff into
gretl is already there. As always, the most time-consuming thing is laying
out the foundations; user-visible changes are but the proverbial tip of
the iceberg. Case in point: the Johansen procedure for cointegration. This
procedure is possibly used hundreds of times a day by applied researchers
all around the world. My personal estimate is that most people use Eviews
or Pc-Give for this. Some others probably use RATS add-ons like CATS
(Juselius) or Malcolm (Mosconi). Maybe someone uses Stata, or custom-made
gauss or Ox routines. Very very few use R's urca module. As we stand,
no-one uses gretl, because the Johansen procedure is only sketched. Still,
the infrastructure is there, and (mostly thanks to Allin) it will take
only a few weeks to have the full Johansen procedure set up. Once we're
there, I have the feeling that the number of users who would choose gretl
would exceed those who use R by several orders of magnitude.
As for competition between R and gretl. In my view, competition is GOOD.
These are not commercial products. These applications are part of the free
software ecosystem, where biodiversity is essential for survival. Contrary
to commercial software, the existence of a competitor produces POSITIVE
externalities (take a look at KDE/Gnome). There is room for R and gretl,
and to spare, just the same way you get Debian, SuSe, Red Hat, Mandriva,
Ubuntu and all the various linux distros.
Riccardo `Jack' Lucchetti
Dipartimento di Economia
Università di Ancona
jack(a)dea.unian.it
http://www.econ.unian.it/lucchetti
6:25 a.m.
New subject: gretl development roadmap
El Sábado, 16 de Julio de 2005 10:18, Jack Lucchetti escribió:
I have heard Ignacio's point many times, but I'm still not convinced. I
had nearly the same conversation with my friend Christine Choirat (of
OxJapi fame) a few months ago. She's an extremely good programmer, and her
line was basically "why re-invent the wheel?". Instead on working on
gretl, I remember she was planning to create a Tcl/Tk wrapper around R
which would mimic Pc-Give as much as possible.
Well, Allin already outlined a few reasons why the wheel is still worthy
of being worked on. First, R is hard.
I admit R is hard, so for a normal undergraduate student Gretl will give a
much more friendly and quick taste of what Econometrics does.
But R is so hard because it is so powerfull!!
I consider myself above the average
econometrician when it comes to computer languages (perhaps immodestly),
and still working with R is a PITA for me. R's syntax is very
idiosyncratic, and either you work with it all the time, or you spend 1
hour doing your work and 9 looking at the manual.
You are right when speaking about the first time, but after a little practice
this proportion improves significatively.
This is most likely a
big problem for someone who is not a full-time econometrician, but it is
likely to be a big hurdle for people who do applied research who want
something that "just works". Remember, these people form the vast majority
of the users of econometrics/statistical packages. Example: I had a
colleague who needed the HP filter. Fairly standard, no? It took me less
time to add it into gretl (with Allin's help) an give him the upgraded
package than it would have taken having him download and set up R,
learning how to import his data, perform the filtering, and going back to
the applications he normally uses. And once this is done, it's done for
everyone. Rather than the fish, you give people the fishing rod.
Sorry Jack, ... this argument is tricky ;-) You already know programming in
C!! Do you know how much time would I have to dedicate to this job?
( I bought the books about C that Allin recommended to me, but I did not pass
the Fahrenheit-Celsius example )
Suppose, once the HP filter in Gretl has been applied, the user needs to apply
another similar smoothing spline (with different parameters, for example).
What he should do?
1- Learn C, study the HP code of gretl and rewrite it conveniently.
2- Learn R and use different values in the command smooth.spline.
3- Ask to you or Allin to program the new code.
A normal user will try to discard point 3 for obvious reasons (May be you
already have enough work). So, what do you think will be more easy and direct
for this user? perhaps the answer may be different for different people, but
I have tried both approaches, and really I think programming in R is a more
direct way.
If the functionality of gretl as a R front-end can be enhanced, that would
be a very cool thing to have (for me at least), but there's no reason to
be tied to it for the less trivial computations.
May be I have been too much strict with this question. I think that running
commands through the entries in the GUI (in the menus) is much more friendly
that running commands in the console. So we should try to fill the menus with
so many elements as the GUI can accept. But I suspect that soon there will be
no room for many more elements than we already have, so gretl will have to
evolution more through the console commands. And this will do the user have
to look more frequently at the manual.
Besides, if you look at
the source, the infrastructure for adding more sophisticated stuff into
gretl is already there.
Yes, this is something that Allin told me, but I had no time to try. Have we
documentation about this?
only a few weeks to have the full Johansen procedure set up. Once
we're
there, I have the feeling that the number of users who would choose gretl
would exceed those who use R by several orders of magnitude.
You are optimistic,... That is good!!
(By the way, do you know Gretl is at the top 5% of sourceforge downloads for
several weeks?)
As for competition between R and gretl. In my view, competition is GOOD.
These are not commercial products. These applications are part of the free
software ecosystem, where biodiversity is essential for survival. Contrary
to commercial software, the existence of a competitor produces POSITIVE
externalities (take a look at KDE/Gnome). There is room for R and gretl,
and to spare, just the same way you get Debian, SuSe, Red Hat, Mandriva,
Ubuntu and all the various linux distros.
I am sure competition is good for the "products", but I think it is not so
good for the users. For example, the R development team decided not to
support the development of an R GUI. So they leave volunteer programmers
to work in a competitive way (see http://www.sciviews.org/_rgui/ ). The result
is that after several years of development the user does not have a decent
graphical interface for R. I am sure that if this efforts had been
coordinated the result would be different.
--
Ignacio Díaz-Emparanza
Dpto. de Economía Aplicada III (Econometría y Estadística)
UPV-EHU
http://www.bl.ehu.es/~etpdihei/
11:46 a.m.
New subject: gretl development roadmap
On Mon, 18 Jul 2005, Ignacio Díaz-Emparanza wrote:
I admit R is hard, so for a normal undergraduate student Gretl will
give a
much more friendly and quick taste of what Econometrics does.
But R is so hard because it is so powerfull!!
I don't dispute that. But I know for a fact that *I* don't like using R.
And indirectly, the proof that other people don't either is in the simple
observation that a package which is so good as R (AND FREE AS IN BEER)
is not yet the de-facto standard in the econometrics community. As you
say, a nice front-end application probably would help a lot in bridging
the gap, but I've never seen one yet. If gretl could do the trick, so be
it. But why relegate gretl to this role, when many things could be done
natively?
> This is most likely a
> big problem for someone who is not a full-time econometrician, but it is
> likely to be a big hurdle for people who do applied research who want
> something that "just works". Remember, these people form the vast
majority
> of the users of econometrics/statistical packages. Example: I had a
> colleague who needed the HP filter. Fairly standard, no? It took me less
> time to add it into gretl (with Allin's help) an give him the upgraded
> package than it would have taken having him download and set up R,
> learning how to import his data, perform the filtering, and going back to
> the applications he normally uses. And once this is done, it's done for
> everyone. Rather than the fish, you give people the fishing rod.
Sorry Jack, ... this argument is tricky ;-) You already know programming in
C!! Do you know how much time would I have to dedicate to this job?
( I bought the books about C that Allin recommended to me, but I did not pass
the Fahrenheit-Celsius example )
Don't overestimate my knowledge of C (Allin can testify!). Besides, in the
HP filter incident, I just found a C routine on the net. I never claimed I
wrote it myself.
> If the functionality of gretl as a R front-end can be enhanced,
that would
> be a very cool thing to have (for me at least), but there's no reason to
> be tied to it for the less trivial computations.
May be I have been too much strict with this question. I think that running
commands through the entries in the GUI (in the menus) is much more friendly
that running commands in the console. So we should try to fill the menus with
so many elements as the GUI can accept. But I suspect that soon there will be
no room for many more elements than we already have, so gretl will have to
evolution more through the console commands. And this will do the user have
to look more frequently at the manual.
Certainly. But gretl's scripting syntax is far easier to grasp to the
casual user than R's syntax is. But again, this is a personal feeling.
Moreover, the most recent additions to the function code make it much
easier to write scripts.
> Besides, if you look at
> the source, the infrastructure for adding more sophisticated stuff into
> gretl is already there.
Yes, this is something that Allin told me, but I had no time to try. Have we
documentation about this?
As Obi-Wan Kenobi would say... "use the source, Luke!" ;-)
No, seriously I think we're a bit lacking in that respect, but the one
person who knows precisely how the source works is already putting so much
effort for the benefit of the community that we can't expect Allin to
write docs too. My advice to you and all the others who would like to have
a peek under the hood is to download the source and tale a look to the
files in the "plugin" subdirectory. Most procedures in there are
reasonably self-contained. That's the way I started, anyway.
> only a few weeks to have the full Johansen procedure set up.
Once we're
> there, I have the feeling that the number of users who would choose gretl
> would exceed those who use R by several orders of magnitude.
You are optimistic,... That is good!!
(By the way, do you know Gretl is at the top 5% of sourceforge downloads for
several weeks?)
That's cool! How do you get these statistics?
> As for competition between R and gretl. In my view, competition
is GOOD.
> These are not commercial products. These applications are part of the free
> software ecosystem, where biodiversity is essential for survival. Contrary
> to commercial software, the existence of a competitor produces POSITIVE
> externalities (take a look at KDE/Gnome). There is room for R and gretl,
> and to spare, just the same way you get Debian, SuSe, Red Hat, Mandriva,
> Ubuntu and all the various linux distros.
I am sure competition is good for the "products", but I think it is not so
good for the users. For example, the R development team decided not to
support the development of an R GUI. So they leave volunteer programmers
to work in a competitive way (see http://www.sciviews.org/_rgui/ ). The result
is that after several years of development the user does not have a decent
graphical interface for R. I am sure that if this efforts had been
coordinated the result would be different.
This is the trouble with all volunteer-based projects. But as you say,
I'm an optimist. Eventually (asymptotically?) what's good for the products
is good for the users. Small-sample bias can be huge, though :-)
Have a nice day,
Riccardo `Jack' Lucchetti
Dipartimento di Economia
Università di Ancona
jack(a)dea.unian.it
http://www.econ.unian.it/lucchetti
3:53 a.m.
New subject: gretl development roadmap
El Lunes, 18 de Julio de 2005 17:46, jack escribió:
> You are optimistic,... That is good!!
> (By the way, do you know Gretl is at the top 5% of sourceforge downloads
> for several weeks?)
That's cool! How do you get these statistics?
The Gretl project page at sourceforge
(http://sourceforge.net/projects/gretl/ )
shows the "activity percentile of the last week" (now: 99.30).
There is also a more complete statistics page at
http://sourceforge.net/project/stats/?group_id=36234&ugn=gretl
--
Ignacio Díaz-Emparanza
Dpto. de Economía Aplicada III (Econometría y Estadística)
UPV-EHU
http://www.bl.ehu.es/~etpdihei/
8:14 a.m.
New subject: gretl development roadmap
On Tue, 19 Jul 2005, Ignacio Díaz-Emparanza wrote:
El Lunes, 18 de Julio de 2005 17:46, jack escribió:
>> You are optimistic,... That is good!!
>> (By the way, do you know Gretl is at the top 5% of sourceforge downloads
>> for several weeks?)
>
> That's cool! How do you get these statistics?
>
The Gretl project page at sourceforge
(http://sourceforge.net/projects/gretl/ )
shows the "activity percentile of the last week" (now: 99.30).
There is also a more complete statistics page at
http://sourceforge.net/project/stats/?group_id=36234&ugn=gretl
Thanks, Ignacio.
Brief comment: the "headline" SF activity percentiles can be a bit
misleading in relation to gretl, since I'm not encouraging the use
of the SF Forums or Tracker apparatus. (I find the format of this
off-Sourceforge Mailman list easier to use.) The SF activity stats
factor in Forum and Tracker activity, but not this list, so they
understate the volume of gretl activity.
Allin Cottrell
7021
days inactive
7025
days old
6 comments
3 participants
participants (3)
-
Allin Cottrell -
Ignacio Díaz-Emparanza -
jack