7:22 a.m.
Hi,
at the conference we agreed that in the future contributed function
packages could use 'foreign' blocks that use R. (Just to be clear: the
big difference here is "contributed". A function package per se can
already use whatever is possible in hansl, for private use. But
currently it won't be accepted to the package server if it relies on a
foreign language.)
We also discussed that the package author must specify the version
requirement for R itself and then specify the needed R packages (along
with their own versions). I would like to propose a way to put this
information in the package's spec file, something like this:
First a list of needed ingredients:
R-packages = R glmnet
The idea is that first 'R' itself needs to be explicitly mentioned,
because it could be that no further R packages are needed. Also, it's
just one character :-)
Then the list of needed version requirements:
R-versions = 3.4.3 1.9-9
Obviously, the ordering here must match that in the 'R-packages' line.
As a refinement, several known-good versions might be specified in a
grouped item enclosed in curly braces:
R-versions = {3.4.3 3.4.4} 1.9-9
From what I understood of our discussion, gretl would not really check
the version requirements, but would use this information to print a
warning message at the 'include' stage.
Something like:
"This user-contributed gretl package requires a local R installation at
the path configured in gretl's preferences. (Needed version: 3.4.3, 3.4.4)
Required R packages: glmnet (version: 1.9-9)
The package will likely fail (with obscure error messages) if any of
those requirements are missing!"
Again, since gretl won't do any real checking, the needed changes do not
seem difficult, mainly a matter of documentation I guess.
Comments welcome.
thanks,
sven
4:51 p.m.
On Mon, 19 Aug 2019, Sven Schreiber wrote:
Hi,
at the conference we agreed that in the future contributed function
packages could use 'foreign' blocks that use R. (Just to be clear: the
big difference here is "contributed". A function package per se can
already use whatever is possible in hansl, for private use. But
currently it won't be accepted to the package server if it relies on a
foreign language.)
We also discussed that the package author must specify the version
requirement for R itself and then specify the needed R packages (along
with their own versions). I would like to propose a way to put this
information in the package's spec file, something like this [...]
Thanks, Sven for the reminder and the proposals, which make sense to
me.
Right now I'm busy finalizing native complex matrix support (to be
paraded out before long), but will get back to this when I can.
Allin
2:07 p.m.
Am 21.08.2019 um 18:51 schrieb Allin Cottrell:
On Mon, 19 Aug 2019, Sven Schreiber wrote:
> We also discussed that the package author must specify the
version
> requirement for R itself and then specify the needed R packages (along
> with their own versions). I would like to propose a way to put this
> information in the package's spec file, something like this [...]
Thanks, Sven for the reminder and the proposals, which make sense to
me.
Right now I'm busy finalizing native complex matrix support (to be
paraded out before long), but will get back to this when I can.
I would like to "ping" on this issue - the idea is that we would need to
have this in place before 2019d is released (or will it be 2020a?), such
that contributed packages can start relying on this extension next year.
Otherwise it couldn't be used before 2020a (or even 2020b) comes out.
In particular -it's not a secret- Artur and I have been working on a
partial wrapper for R's glmnet package. It's not finished yet, but
probably would be ready this Winter.
thanks
sven
3:57 p.m.
On Sat, 26 Oct 2019, Sven Schreiber wrote:
In particular -it's not a secret- Artur and I have been working
on a
partial wrapper for R's glmnet package. It's not finished yet, but
probably would be ready this Winter.
That would be excellent.
However, I consider it of the highest importance to have a native
implementation of LASSO (at least, but elastic net would be nice too)
ready as soon as time allows. Allin and I talked about it briefly in June.
I guess we could start thinking about it.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
5:10 p.m.
Am 26.10.19 um 17:57 schrieb Riccardo (Jack) Lucchetti:
On Sat, 26 Oct 2019, Sven Schreiber wrote:
> In particular -it's not a secret- Artur and I have been working on a
> partial wrapper for R's glmnet package. It's not finished yet, but
> probably would be ready this Winter.
That would be excellent.
However, I consider it of the highest importance to have a native
implementation of LASSO (at least, but elastic net would be nice too)
ready as soon as time allows. Allin and I talked about it briefly in
June. I guess we could start thinking about it.
Having a built-in LASSO would be cool of course. Some time ago I was
looking for existent C-based implementations. On github one find some
existent libraries. Of course: No idea how well and accurate they work
and whether stuff is properly tested. But maybe there exists some
well-written library similar to libsvm.
Still, the glmnet library offers support for several types of models:
gaussian, multinomial, cox etc. Not sure whether you plan to support all
of these - I guess not as this involves a lot of work. Hence, the the
glmnet wrapper could still support _additional _ model types.
Best,
Artur
1:41 a.m.
On Sat, 26 Oct 2019, Artur Tarassow wrote:
Am 26.10.19 um 17:57 schrieb Riccardo (Jack) Lucchetti:
> On Sat, 26 Oct 2019, Sven Schreiber wrote:
>
>> In particular -it's not a secret- Artur and I have been working on a
>> partial wrapper for R's glmnet package. It's not finished yet, but
>> probably would be ready this Winter.
>
> That would be excellent.
>
> However, I consider it of the highest importance to have a native
> implementation of LASSO (at least, but elastic net would be nice too) ready
> as soon as time allows. Allin and I talked about it briefly in June. I
> guess we could start thinking about it.
Having a built-in LASSO would be cool of course. Some time ago I was looking
for existent C-based implementations. On github one find some existent
libraries. Of course: No idea how well and accurate they work and whether
stuff is properly tested. But maybe there exists some well-written library
similar to libsvm.
Still, the glmnet library offers support for several types of models:
gaussian, multinomial, cox etc. Not sure whether you plan to support all of
these - I guess not as this involves a lot of work. Hence, the the glmnet
wrapper could still support _additional _ model types.
It so happens that over the last few days I've been reviewing where
Jack and I got to with lasso this past summer, with a possible
function package in mind.
I'm currently testing against glmnet, trying to figure out where
differences are coming from and whether there's anything wrong with
the method we're using (the Alternating Direction Method of
Multipliers, from Stephen Boyd et al.) I think what we have is
sound, but the confusing thing is comparing lambdas: the Boyd et al
algorithm takes as argument the fraction, s, of the lambda value
lmax that drives all coefficients to zero while glmnet (seems to)
take the "raw" lambda value. I'm not totally clear on how lmax
should be calculated and/or exactly what glmnet is doing.
Anyway, seems to me a reliable lasso implementation in hansl for
gaussian linear models would be a useful first step. Right now I'm
not too bothered about trying to emulate elnet's lasso/ridge hybrid,
or the non-gaussian variants supported by glmnet.
Allin
11:10 a.m.
Am 27.10.2019 um 02:41 schrieb Allin Cottrell:
On Sat, 26 Oct 2019, Artur Tarassow wrote:
>
> Having a built-in LASSO would be cool of course.
> Still, the glmnet library offers support for several types of
models:
> gaussian, multinomial, cox etc. Not sure whether you plan to support
> all of these - I guess not as this involves a lot of work. Hence, the
> the glmnet wrapper could still support _additional _ model types.
Anyway, seems to me a reliable lasso implementation in hansl for
gaussian linear models would be a useful first step. Right now I'm not
too bothered about trying to emulate elnet's lasso/ridge hybrid, or the
non-gaussian variants supported by glmnet.
I agree with both of you, which means having both options is nice.
That's why it would be useful to extend the package "spec" file syntax
with the R-related stuff. Again, AFAICS this just means that gretl
recognizes these new lines and prints out a warning message. The rest is
basically the job of the package moderators.
cheers
sven
11:48 p.m.
On Sat, 26 Oct 2019, Sven Schreiber wrote:
I would like to "ping" on this issue - the idea is that we
would need to
have this in place before 2019d is released (or will it be 2020a?), such
that contributed packages can start relying on this extension next year.
Otherwise it couldn't be used before 2020a (or even 2020b) comes out.
I'm pretty sure we'll have a 2019d, though that's by the way.
I've begun thinking about representation of R requirements in the
spec file for a contributed package. Seems to me the information
should probably be in a single line, something like
R-depends = R >= 3.4.3, glmnet >= 1.9-9
If we're just going to print the stuff to the right of the first '='
as part of a warning when a user fires up a relevant package, easy
enough. Parsing and checking it would not be so easy. The 3-number R
versions we could handle, but R package versioning is a mess: a
hodge-podge of Major.Minor, Major.Minor.Patch and Major.Minor-Patch
(maybe plus other forms?).
Should we allow for listing multiple known-good versions (of R
itself or R packages)? I think I'd prefer just the ">=" notation,
with the proviso (in the warning text) that we can't guarantee that
later versions will still work, although that should be a reasonable
expectation.
Allin
7:11 a.m.
Am 28.10.2019 um 00:48 schrieb Allin Cottrell:
I've begun thinking about representation of R requirements in the
spec
file for a contributed package. Seems to me the information should
probably be in a single line, something like
R-depends = R >= 3.4.3, glmnet >= 1.9-9
If we're just going to print the stuff to the right of the first '=' as
part of a warning when a user fires up a relevant package, easy enough.
Parsing and checking it would not be so easy.
I don't think gretl should be responsible for checking the requirements.
At least that was my understanding of the discussion.
Should we allow for listing multiple known-good versions (of R itself
or
R packages)? I think I'd prefer just the ">=" notation, with the
proviso
(in the warning text) that we can't guarantee that later versions will
still work, although that should be a reasonable expectation.
I would be fine with that, although I do believe it does happen that
later versions will break something. But that's not a big problem IMO.
thanks
sven
9:32 p.m.
On Sat, 26 Oct 2019, Sven Schreiber wrote:
Am 21.08.2019 um 18:51 schrieb Allin Cottrell:
> On Mon, 19 Aug 2019, Sven Schreiber wrote:
>> We also discussed that the package author must specify the version
>> requirement for R itself and then specify the needed R packages (along
>> with their own versions). I would like to propose a way to put this
>> information in the package's spec file, something like this [...]
>
> Thanks, Sven for the reminder and the proposals, which make sense to
> me.
>
> Right now I'm busy finalizing native complex matrix support (to be
> paraded out before long), but will get back to this when I can.
I would like to "ping" on this issue - the idea is that we would need to
have this in place before 2019d is released (or will it be 2020a?), such
that contributed packages can start relying on this extension next year.
Otherwise it couldn't be used before 2020a (or even 2020b) comes out.
New in git:
* The function package DTD now includes an optional element named
"R-depends" to record R dependency information.
* This is now handled in the process of "compiling" a gfn file via
use of a .spec file. And it's documented in the current iteration of
the Function Package doc.
* When a package with an R dependency is loaded, a message is
printed alerting the user to this fact, with specifics on OK
versions.
At present the R information cannot be set via the packaging GUI,
but that can be added. I've settled on a simple format for the
dependency info, which should be relatively easy to parse if at some
future point we want to implement checking that the requirements are
satisfied (something we do not yet attempt). Example:
R-depends = R 3.5.3 glmnet 2.0-18
This means that the package requires R and R's glmnet, and is known
to work OK with R version 3.5.3 plus glmnet 2.0-18. The possibility
is left open that it might work with earlier or later versions of
both -- the key point is that it's tested OK with this combination.
More generally, the required format is:
R <version>
followed on the same line by zero or more instances of
<R-pkg> <version>
as needed, all separated by spaces.
Allin
10:26 a.m.
Am 30.10.2019 um 22:32 schrieb Allin Cottrell:
New in git:
* The function package DTD now includes an optional element named
"R-depends" to record R dependency information.
* This is now handled in the process of "compiling" a gfn file via use
of a .spec file. And it's documented in the current iteration of the
Function Package doc.
* When a package with an R dependency is loaded, a message is printed
alerting the user to this fact, with specifics on OK versions.
At present the R information cannot be set via the packaging GUI, but
that can be added.
Thanks Allin, I think the latter limitation is perfectly OK.
(something we do not yet attempt). Example:
R-depends = R 3.5.3 glmnet 2.0-18
This means that the package requires R and R's glmnet, and is known to
work OK with R version 3.5.3 plus glmnet 2.0-18. The possibility is left
open that it might work with earlier or later versions of both -- the
key point is that it's tested OK with this combination.
But if it's not meant in a >= sense, then the question arises how to
give several versions which are known to work. I can see the point that
it "usually" will work with newer versions, and so that information is
perhaps not super important. But if you want to convey the meaning that
it is actually tested, perhaps the package author did really test on
multiple versions?
(Putting several space-separated versions in parentheses of some kind
also seems feasible to parse.)
thanks
sven
12:07 p.m.
On Thu, 31 Oct 2019, Sven Schreiber wrote:
Am 30.10.2019 um 22:32 schrieb Allin Cottrell:
> New in git:
>
> * The function package DTD now includes an optional element named
> "R-depends" to record R dependency information.
>
> * This is now handled in the process of "compiling" a gfn file via use
> of a .spec file. And it's documented in the current iteration of the
> Function Package doc.
>
> * When a package with an R dependency is loaded, a message is printed
> alerting the user to this fact, with specifics on OK versions.
>
> At present the R information cannot be set via the packaging GUI, but
> that can be added.
Thanks Allin, I think the latter limitation is perfectly OK.
> (something we do not yet attempt). Example:
>
> R-depends = R 3.5.3 glmnet 2.0-18
>
> This means that the package requires R and R's glmnet, and is known to
> work OK with R version 3.5.3 plus glmnet 2.0-18. The possibility is left
> open that it might work with earlier or later versions of both -- the
> key point is that it's tested OK with this combination.
But if it's not meant in a >= sense, then the question arises how to
give several versions which are known to work. I can see the point that
it "usually" will work with newer versions, and so that information is
perhaps not super important. But if you want to convey the meaning that
it is actually tested, perhaps the package author did really test on
multiple versions?
(Putting several space-separated versions in parentheses of some kind
also seems feasible to parse.)
OK, we could go that way. But my preference is to keep this line as
simple as possible. I'm thinking that if a package author has tested
successfully with multiple R/package versions, she should just record
the earliest known-good versions in the R-depends line. (These may or
may not be the earliest versions that actually work, since we don't
expect a package writer to back-track through R and R-package versions
to find the exact point at which they're too old.) But the package
help text might usefully add information about any other known-good
versions.
Allin
12:11 p.m.
Am 31.10.2019 um 13:07 schrieb Allin Cottrell:
On Thu, 31 Oct 2019, Sven Schreiber wrote:
> But if it's not meant in a >= sense, then the question
arises how to
> give several versions which are known to work. I can see the point that
> it "usually" will work with newer versions, and so that information is
> perhaps not super important. But if you want to convey the meaning that
> it is actually tested, perhaps the package author did really test on
> multiple versions?
> (Putting several space-separated versions in parentheses of some kind
> also seems feasible to parse.)
OK, we could go that way. But my preference is to keep this line as
simple as possible. I'm thinking that if a package author has tested
successfully with multiple R/package versions, she should just record
the earliest known-good versions in the R-depends line. (These may or
may not be the earliest versions that actually work, since we don't
expect a package writer to back-track through R and R-package versions
to find the exact point at which they're too old.) But the package
help text might usefully add information about any other known-good
versions.
OK, fair enough. The most important thing is to have the R-possibility
in place. Thanks!
-sven
1918
days inactive
1991
days old
12 comments
4 participants
participants (4)
-
Allin Cottrell -
Artur Tarassow -
Riccardo (Jack) Lucchetti -
Sven Schreiber