Hi, I want to raise a "policy question". Currently contributed function packages can decide "for themselves" whether they want to be added somewhere in the gretl menus. IIRC this was borrowed from the gretl add-ons. For the outsider/user this blurs the distinction between core gretl and the contributions, which is exactly the purpose of the mechanism. However, there is a difference in the amount of testing that goes into a functionality in core gretl compared to the testing that a function package receives. (I'm talking about the testing done by the code authors, not about the quite superficial checks that Allin, Jack, and I are doing when moving a submitted package to the public server. And I am also talking about packages written by myself, this is not meant as criticizing others.) So the problem is that to the user package bugs would look like gretl bugs, which makes gretl look less reliable than it is. My first suggestion therefore is to include the "author" field not only in the package list in the "on server" window, but also in the list window displaying the locally installed packages. This should make it more transparent that these are _contributed_ packages, and not core gretl. (Also the ego of the authors might benefit a little.) Secondly, my proposal is that before a package is allowed to appear in the menus it would need to pass some additional checks. For example, this check could be provided by replicating in the example script some known-good result. BTW, there is a real-world experience behind my suggestion. Of course this policy change would not prevent all bugs; for example I remember a bug affecting (some) results in the SVAR addon, and that addon was certainly tested quite thoroughly. But still I think it could help. So -- what do you think? thanks, sven _______________________________________________ Gretl-devel mailing list Gretl-devel(a)lists.wfu.edu http://lists.wfu.edu/mailman/listinfo/gretl-devel

On Thu, 19 Jan 2017, Sven Schreiber wrote: > Hi, > > I want to raise a "policy question". Currently contributed function packages > can decide "for themselves" whether they want to be added somewhere in the > gretl menus. IIRC this was borrowed from the gretl add-ons. > > For the outsider/user this blurs the distinction between core gretl and the > contributions, which is exactly the purpose of the mechanism. > > However, there is a difference in the amount of testing that goes into a > functionality in core gretl compared to the testing that a function package > receives. (I'm talking about the testing done by the code authors, not about > the quite superficial checks that Allin, Jack, and I are doing when moving a > submitted package to the public server. And I am also talking about packages > written by myself, this is not meant as criticizing others.) > > So the problem is that to the user package bugs would look like gretl bugs, > which makes gretl look less reliable than it is. > My first suggestion therefore is to include the "author" field not only in > the package list in the "on server" window, but also in the list window > displaying the locally installed packages. This should make it more > transparent that these are _contributed_ packages, and not core gretl. (Also > the ego of the authors might benefit a little.) I don't see why we shouldn't do this. > Secondly, my proposal is that before a package is allowed to appear in the > menus it would need to pass some additional checks. For example, this check > could be provided by replicating in the example script some known-good > result. I don't like this very much. When you install a package, you give it explicit permission to show up in the main GUI somewhere, so you can't claim "nobody told me". And besides, giving access to the menus as if it was some badge of honour that you earn by proving your package replicates some known benchmark would be (a) practically impossible in some cases (think eg of Ignacio's HEGY package) (b) very burdensome in terms of QA assurance (c) very easy to circumvent (edit the package - add the menu property - save the package). > BTW, there is a real-world experience behind my suggestion. Of course this > policy change would not prevent all bugs; for example I remember a bug > affecting (some) results in the SVAR addon, and that addon was certainly > tested quite thoroughly. But still I think it could help. One thing that we could do, in order to inform the user on the degree of testing a certain package has undergone, is to give package authors the possibility of writing a short working paper (in the "gretl wp" collection) where replication of known examples is performed, like Claudia and myself did for DPB, and advertise that in the package description. ------------------------------------------------------- Riccardo (Jack) Lucchetti Dipartimento di Scienze Economiche e Sociali (DiSES) Università Politecnica delle Marche (formerly known as Università di Ancona) r.lucchetti(a)univpm.it http://www2.econ.univpm.it/servizi/hpp/lucchetti ------------------------------------------------------- ------------------------------------------------------------------------ _______________________________________________ Gretl-devel mailing list Gretl-devel(a)lists.wfu.edu http://lists.wfu.edu/mailman/listinfo/gretl-devel

On Thu, 19 Jan 2017, Sven Schreiber wrote:

> My first suggestion therefore is to include the "author" field not > only in the package list in the "on server" window, but also in the > list window displaying the locally installed packages. This should > make it more transparent that these are _contributed_ packages, and > not core gretl. (Also the ego of the authors might benefit a little.) I don't see why we shouldn't do this.

One thing that we could do, in order to inform the user on the degree of testing a certain package has undergone, is to give package authors the possibility of writing a short working paper (in the "gretl wp" collection) where replication of known examples is performed, like Claudia and myself did for DPB, and advertise that in the package description.

Am 19.01.2017 um 15:32 schrieb Riccardo (Jack) Lucchetti: > On Thu, 19 Jan 2017, Sven Schreiber wrote: > >> Secondly, my proposal is that before a package is allowed to appear in >> the menus it would need to pass some additional checks. For example, >> this check could be provided by replicating in the example script some >> known-good result. > > I don't like this very much. When you install a package, you give it > explicit permission to show up in the main GUI somewhere, so you can't > claim "nobody told me". You have a point about the explicit menu permission. And maybe the relevant threshold shouldn't have to do with the menus. But I think we agree that the level of quality assurance is different between core gretl and the contributed packages. And I believe it should be communicated to users that they're taking a somewhat higher risk with contributed packages.

In the world of R, my impression is that users are aware that they're leaving core R when they're using packages from CRAN. For the gretl world I'm having doubts about this, perhaps because of the extremely nice GUI integration of packages...

[...] My first suggestion therefore is to include the "author" field not only in the package list in the "on server" window, but also in the list window displaying the locally installed packages. This should make it more transparent that these are _contributed_ packages, and not core gretl. (Also the ego of the authors might benefit a little.)

Secondly, my proposal is that before a package is allowed to appear in the menus it would need to pass some additional checks. For example, this check could be provided by replicating in the example script some known-good result.

On Thu, 19 Jan 2017, Allin Cottrell wrote: >> Secondly, my proposal is that before a package is allowed to appear in the >> menus it would need to pass some additional checks. For example, this >> check could be provided by replicating in the example script some >> known-good result. > > I agree with Jack that this would be awkward to implement. However, I also > agree with you that including replication of known good results in a > package is something that should be encouraged, even if not required. It > might be worth adding a field to the gfn specification to specify a > replication/test script (which I guess could be the same thing as the > sample script but might be distinct). That would make it possible to detect > right off if a given package has such a thing. In most cases, replication is a non-trivial task, that needs to be descrbed in a reasonable amount of detail, if the package is of some complexity. I remain convinced that a pdf file describing exactly what is being done and how would be essential in many cases, if not most.

Am 19.01.2017 um 19:50 schrieb Allin Cottrell: > On Thu, 19 Jan 2017, Riccardo (Jack) Lucchetti wrote: >> In most cases, replication is a non-trivial task, that needs to be >> descrbed in a reasonable amount of detail, if the package is of some >> complexity. I remain convinced that a pdf file describing exactly what >> is being done and how would be essential in many cases, if not most. > > OK, I agree that'll be true in many cases. I don't think it is currently true that the majority of packages are so complex. In any case, what's the way forward? Jack's opinion seems to be, rely on the community to spot and report bugs.

He has a point, but I believe that the barriers to bug reporting are still too high. Nobody is going to join a mailing list nowadays just to report what might or might not be a single bug.

Of course we do have the bug tracker on sourceforge, but it's not very visible. First, it's not mentioned on the gretl homepage. Secondly, even if you understand the relatively cryptic heading "gretl project page" and follow that to https://sourceforge.net/projects/gretl/, it's still not obvious where to report bugs -- you have to click on "Tickets" first. Perhaps we should have a direct link from the front page to https://sourceforge.net/p/gretl/bugs/ ?

And please, can we finally remove the link to the dead gretl wiki from the front page?

> And please, can we finally remove the link to the dead gretl wiki from the > front page? OK, done. I've substituted a link to the gretl working papers. Jack, please correct me if the link I've inserted is not the preferred one.

On Fri, 20 Jan 2017, Sven Schreiber wrote:

> Of course we do have the bug tracker on sourceforge, but it's not very > visible. First, it's not mentioned on the gretl homepage. It is actually mentioned, with a direct link, but I've now made it more prominent.

> And please, can we finally remove the link to the dead gretl wiki from > the front page? OK, done. I've substituted a link to the gretl working papers.