On Mon, 6 Jul 2009, Riccardo (Jack) Lucchetti wrote: Yes, agreed on both counts. Yes, indeed. I've been thinking about that too. The first point to make is that trying to interpret a "word" as an R function name is the "last resort" in genlex.c. So any built-in, or user-defined, gretl function will mask an R function of the same name. My first thought on this is to the same effect but a little simpler: we could mark off R functions (either built-in, or defined via "foreign" blocks within gretl) by using the prefix "R_". So if you wanted to invoke the R "choose" function you'd have to invoke it as "R_choose(args)". (Internally, we'd see the "R_" and pass the remainder of the word to the new function get_R_function_by_name().) This would reliably avert clashes with gretl built-in functions since gretl uses lower-case identifiers exclusively -- and I don't envisage changing that since I hate mixed-case function names! It does mean that if someone defines a user function "R_foo" it will mask any "foo" function that may exist in R, but that doesn't seems like a big problem to me -- particularly if the manual mentions this point. Yes, it does raise a portability issue. We do have some of those already: for example, the gretl command "mpols" will work only if gretl is linked against (the optional) libgmp. There's also the business of the optional third-party programs TRAMO/SEATS and X-12-ARIMA. At present the functionality of these programs is available only via the GUI, but it probably should be available via script. There are some other issues too. libR is not an "ordinary" library: by plugging into its API you basically "buy into" an R session. And this can be quite lumpy. R seems to make liberal use of the evil "longjmp" function (C's non-local goto -- and you thought that "goto" was bad!). So, for example, the innocuous-looking R function "findFun" crashes the caller on failure. Similarly, a call to "Rf_initEmbeddedR" will abort the caller if the environment is not set up correctly for R. I've tried to guard against both of these issues in recent changes in gretl_foreign.c, but I'm sure there are more. In addition, if you run a gretl/Rlib session under valgrind valgrind --tool=memcheck --leak-check=full --show-reachable=yes you'll see that libR wants to allocate at least 10 MB before it does _anything_. Christoph gave an example function, defined in R, that took a scalar argument and returned its argument plus 1. Fine, R will do that for you if you give it 10 MB of RAM and a several dozen function calls. (OK, any library uses resources; I'm just saying that the libR approach is not a magic bullet for saving on initialization costs.) Allin.

On Mon, 6 Jul 2009, Allin Cottrell wrote: I like this. The only possible problem I can see is parsing the string correctly when a dot is there, as in R_as.matrix() True. And you can add LaTeX on top of that if you want. But you'll agree the ones you mention are rather special cases: mpols is definitely not something you'd normally use for regressions, and the TRAMO/SEATS-X12 stuff is hard to use inadvertently, so to speak. My concern was coming from the idea that if you start mixing native and R functions in your scripts, to the point when it's hard to tell them apart, as in your choose() example, then one day you take your script to another machine where R isn't installed and, bang!, your script is useless. [...] Chris' work and yours have been splendid. Integration with R is now much, much better than it was. Thanks! Riccardo (Jack) Lucchetti Dipartimento di Economia Università Politecnica delle Marche r.lucchetti(a)univpm.it http://www.econ.univpm.it/lucchetti

On Tue, 7 Jul 2009, Talha Yalta wrote: No-one is forcing anyone to use R from within gretl. But you'll agree with me that having the possibility of interacting in the smoothest possible way with R is 1) nice to have: hey, this is free software; if R is better than gretl at something, why not use it? 2) cool, as gretl does R, but R doesn't do gretl :-) 3) attractive for people who like R but would like a friendlier environment 4) totally orthogonal to gretl's future development plans. So I can't see what the downside is. Riccardo (Jack) Lucchetti Dipartimento di Economia Università Politecnica delle Marche r.lucchetti(a)univpm.it http://www.econ.univpm.it/lucchetti

I agree with the above points. I also agree with the concerns that you expressed regarding function names and portability. All I am saying deciding on how to integrate R with gretl is an important decision and requires a policy on whether using R within gretl is a)- someting nice to have b)- encouraged c)- nice to have but in fact discouraged Depending on the answer above, one can implement this by for example a)- creating a function or a program that can automatically translate a gretl script to an R script and vice versa b)- making it possible to freely combine R functions with gretl functions in gretl scripts c)- making it possible to use R fully but within a well-defined boundary such as with a construct like begin R <R code> end R And believe me this is not someting "totally orthogonal" to gretl's future development. There can be important secondary and tertiary effects that we can or cannot foresee right now. -- “Remember not only to say the right thing in the right place, but far more difficult still, to leave unsaid the wrong thing at the tempting moment.” - Benjamin Franklin (1706-1790) --

On Tue, 7 Jul 2009, Talha Yalta wrote: I don't really think we need a policy here. Even from a political standpoint (which is not my primary concern anyway), we're not even mixing licences, since both gretl and R are GPL. There may be gretl users who don't even know what R is, people who use it sporadically (like myself), others who may use gretl just as an R frontend. Again, I can't see any problem with any of these attitudes. In my view, gretl's aim should be to make applied econometrics as practical and efficient as possible. If this involves borrowing stuff from other projects, so be it. Let me remind that we've been doing this forever, internally (gretl's source is full of stuff adapted from R, netlib, gnumeric etcetera). It's called "division of labour" :-) The only drawback to this idea is that encapsulating code from other projects may be so impractical to outweigh the benefits. I had a conversation recently with Christian Brownlees, the author of dynamo (http://dynamo.sourceforge.net/), who argued that he'd like to integrate dynamo into gretl, but that implies a dependency on libgsl, which is not on our agenda, at least in the short term (see http://lists.wfu.edu/pipermail/gretl-users/2009-June/003351.html). But in principle cooperation between projects is IMHO the way to go. Well, we've had (c) for a while, as the "foreign" command. For all I know, people are quite happy with it and nothing tragic has happened so far. Riccardo (Jack) Lucchetti Dipartimento di Economia Università Politecnica delle Marche r.lucchetti(a)univpm.it http://www.econ.univpm.it/lucchetti

On Tue, 7 Jul 2009, Sven Schreiber wrote: Yes, that's a real exercise in ingenuity, formulating a google string that has a chance of success! s/R/CRAN/ sometimes works. Allin.

Riccardo (Jack) Lucchetti wrote: The question of how to implement this new functionality in gretl is really important!! We should weight the pros and cons of each approach, in order not to end up with a badly specified interface. In the following you will find my proposal: I created the libr-functionality because I was unhappy with the foreign blocks. When I was writing R code I had to write gretl stubs for every R function, in order to be able to pass parameters and return from and into R. The new libr interface now makes this obsolete. I think that most of the time you don't want use directly a R function, because some of the more complex functions return variable types which won't be supported in the near future by gretl (lists, s4 objects). So I think that in general we should not make all R functions avaiable to gretl, as a lot cannot be used from within gretl. In order to make these function usable from within gretl, you have to define a wrapper function which converts the arguments and returns the more simple datatypes that gretl understands.These wrappers should be defined in a foreign block, using "language=libr". And to make them accesible from within gretl, we could enhance the gretl.export() routines. If gretl.export is called from within a foreign libr-block and a function is supplied as argument, the programm will make this function avaiable for gretl scripts. Example: <script> nulldata 10 foreign language=RLib func1<-function(q) { z=sum(q[1,]) invisible(z) } func2<-function(q) { z=mean(q[1,]) invisible(z) } gretl.export(func1) end foreign scalar a=1 matrix qq=ones(2,2) #the statement below will work scalar b=func1(qq) #func2 has not been exported->error scalar b=func2(qq) #the same is true for chose scalar b=choose(2) </script> With this approach we won't pollute the gretl namespace. The foreign blocks will be executed when the file is included, so you know straight from the beginning if it runs. Christoph

On Tue, 7 Jul 2009, chris wrote: Yup, I think we're all agreed on that. Yes, it's very nice to be able to send inputs, and grab output, directly. True, but I don't see any harm in exposing things like "choose" and other such relatively simple built-ins. That's certainly worth thinking about. We'll have to come back to this. Allin.