11:21 a.m.
Hi,
playing around with the 'store' command, I came across the following (on
1.6.0 self-compiled on ubuntu edgy):
* store saves data in ~, even though ~/gretl is activated in prefs
* with data from example file data2-3.gdt (annual), store saves dates as
literally "'1995" (including the double quotes); reopening works fine,
but why all these quotes, including the single quote?
* store --csv saves first column with heading "obs", although in the
guide it says that for reading the column must have header "date" ??
(well actually haven't double-checked that last assumption, sorry, am in
a rush) But reopening with ascii works, so just a documentation bug?
* gui importing from csv seems to presuppose .csv extension, must use
ascii import for csv files with different extension -- is that necessary?
* 'open' seems to rely on heuristics only; why no possibility to specify
e.g. --csv as in store?
Sorry if anything has been obsoleted in newer versions,
Sven
2:52 p.m.
On Fri, 15 Dec 2006, Sven Schreiber wrote:
playing around with the 'store' command, I came across the
following (on 1.6.0 self-compiled on ubuntu edgy):
* store saves data in ~, even though ~/gretl is activated in prefs
The notion is that "store" is an "expert" command: the filename
used is exactly what you type on the command line, so if it's not
an absolute path it goes into the present working directory. This
should be documented, no doubt.
* with data from example file data2-3.gdt (annual), store saves
dates as literally "'1995" (including the double quotes);
reopening works fine, but why all these quotes, including the
single quote?
This may be redundant with annual data, but as I recall it's an
attempt to avoid breakage when saving, e.g., quarterly or monthly
data for use in a spreadsheet. Excel will do weird and wonderful
things with, say, "1975:01", unless a single quote is prepended to
indicate that it's a literal string.
* store --csv saves first column with heading "obs",
although in the
guide it says that for reading the column must have header "date" ??
"obs" and "date" are equally acceptable in this context.
* gui importing from csv seems to presuppose .csv extension,
must use ascii import for csv files with different extension --
is that necessary?
No, probably not.
* 'open' seems to rely on heuristics only; why no possibility
to
specify e.g. --csv as in store?
Because, generally speaking, gretl is smarter than most users when
it comes to figuring out the format of imported data! ;-) But
seriously, what useful information could you give gretl by saying
"--csv"? Gretl will have to check the format of the data anyway.
Allin.
11:25 p.m.
Allin Cottrell schrieb:
On Fri, 15 Dec 2006, Sven Schreiber wrote:
>
> * store saves data in ~, even though ~/gretl is activated in prefs
The notion is that "store" is an "expert" command: the filename used
is
exactly what you type on the command line, so if it's not an absolute
path it goes into the present working directory. This should be
documented, no doubt.
Which (environment?) variable determines that directory then? Or related
to that: How can I find out in a script where the user directory (which
is set in gretl's preferences) is to use that value?
> * with data from example file data2-3.gdt (annual), store saves dates
> as literally "'1995" (including the double quotes); reopening works
> fine, but why all these quotes, including the single quote?
This may be redundant with annual data, but as I recall it's an attempt
to avoid breakage when saving, e.g., quarterly or monthly data for use
in a spreadsheet. Excel will do weird and wonderful things with, say,
"1975:01", unless a single quote is prepended to indicate that it's a
literal string.
Well gretl isn't entirely consistent here; consider what a 'store
test.csv LRM --csv' gives me for the quarterly Danish example data:
obs,LRM
"1974Q1",11.63255023
"1974Q2",11.60415248
"1974Q3",11.58152067
"1974Q4",11.60184741
"1975Q1",11.58630490
<etc.>
I have two problems with that:
1) Ok, "1974Q1" is a string and thus comes with quotes. But so are
"obs"
and "LRM", yet they don't have quotes around them.
2) For annual data you get "'1960" as a spreadsheet workaround, but for
quarterly you don't get the single quote. Actually, although I
understand your Excel argument for stuff with a ":" in it, I don't
understand why that would be relevant for simple year integers in annual
data.
All this makes parsing gretl's csv files error-prone IMHO. I propose to
do one of the following two things: Either enclose _all_ strings in
double quotes, and have the enclosed-but-leading single quote for all
date values; (X)OR remove _all_ quotes, and change the separator ':' to
'M', 'W', 'D' according to the data frequency, in analogy to the
'Q' for
quarterly data. What do you think?
> * gui importing from csv seems to presuppose .csv extension, must use
> ascii import for csv files with different extension -- is that necessary?
No, probably not.
Then I would suggest to mimic the behavior of ASCII import, or possibly
merge them anyway, just like in the 'open' command.
> * 'open' seems to rely on heuristics only; why no possibility to
> specify e.g. --csv as in store?
Because, generally speaking, gretl is smarter than most users when it
comes to figuring out the format of imported data! ;-) But seriously,
what useful information could you give gretl by saying "--csv"? Gretl
will have to check the format of the data anyway.
Ok, I take your point, until gretl screws up, then I will come back to
you ;-)
thanks,
sven
1:05 a.m.
On Sat, 16 Dec 2006, Sven Schreiber wrote:
Allin Cottrell schrieb:
> On Fri, 15 Dec 2006, Sven Schreiber wrote:
>>
>> * store saves data in ~, even though ~/gretl is activated in prefs
>
> The notion is that "store" is an "expert" command: the filename
used is
> exactly what you type on the command line, so if it's not an absolute
> path it goes into the present working directory. This should be
> documented, no doubt.
Which (environment?) variable determines that directory then?
PWD, on *nix systems (I don't know about Windows).
Or related to that: How can I find out in a script where the
user directory (which is set in gretl's preferences) is to use
that value?
Urgh. Why do you need that? I suppose it could be exported to
the environment if need be, but right now there's no way to do
that, unless perhaps by clever use of the "!" shell command.
(In relation to storing time-series data as text/CSV)
Well gretl isn't entirely consistent here; consider what a
'store
test.csv LRM --csv' gives me for the quarterly Danish example data:
obs,LRM
"1974Q1",11.63255023
"1974Q2",11.60415248 ...
I have two problems with that:
1) Ok, "1974Q1" is a string and thus comes with quotes. But so are
"obs"
and "LRM", yet they don't have quotes around them.
But they're bound to be interpreted as strings. Quotes are a
no-op.
2) For annual data you get "'1960" as a spreadsheet
workaround, but for
quarterly you don't get the single quote. Actually, although I
understand your Excel argument for stuff with a ":" in it, I don't
understand why that would be relevant for simple year integers in annual
data.
It wouldn't. There's an accretion of kludges going on here.
All this makes parsing gretl's csv files error-prone IMHO. I
propose to do one of the following two things: Either enclose
_all_ strings in double quotes, and have the
enclosed-but-leading single quote for all date values; (X)OR
remove _all_ quotes, and change the separator ':' to 'M', 'W',
'D' according to the data frequency, in analogy to the 'Q' for
quarterly data. What do you think?
Sounds reasonable. One consideration is the possibility of
"round-tripping". That is, ideally if you export gretl data to
CSV then read them back into gretl, the time-series information
should be preserved. I'm not sure if that's the case at present,
but it's something I've thought of (at least intermittently) when
messing with the exporter/importer code.
>> * gui importing from csv seems to presuppose .csv extension,
>> must use ascii import for csv files with different extension
>> -- is that necessary?
>
> No, probably not.
Then I would suggest to mimic the behavior of ASCII import, or possibly
merge them anyway, just like in the 'open' command.
One possibility I've had in mind for a while is merging everything
under "open" (scrap the "import" command and associated menu
items). Unfortunately, other issues (such as implementing
Arellano-Bond or fixing the behaviour of sub-sampling commands
within user-defined functions) have got in the way ;-)
Allin.
8:11 p.m.
Allin Cottrell schrieb:
On Sat, 16 Dec 2006, Sven Schreiber wrote:
> Allin Cottrell schrieb:
>> On Fri, 15 Dec 2006, Sven Schreiber wrote:
>
>> The notion is that "store" is an "expert" command: the
filename used is
>> exactly what you type on the command line, so if it's not an absolute
>> path it goes into the present working directory. This should be
>> documented, no doubt.
> Or related to that: How can I find out in a script where the
user
> directory (which is set in gretl's preferences) is to use that value?
Urgh. Why do you need that? I suppose it could be exported to the
environment if need be, but right now there's no way to do that, unless
perhaps by clever use of the "!" shell command.
Well if 'store' is expert-ish, then it needs some expert information ;-)
Anyway, I would prefer if 'store' simply uses the gretl preference
setting of the user directory. At least that is what I expected first.
Now everything is clobbering the top level of my home dir. But creating
directories in a gretl script would only work via the shell AFAICS, and
that means cross-platform problems. I mean the only change would be the
point of reference for relative paths, now it's externally determined,
and it would become gretl-determined. I think the latter is better.
<snip>
> All this makes parsing gretl's csv files error-prone IMHO. I propose
> to do one of the following two things: Either enclose _all_ strings in
> double quotes, and have the enclosed-but-leading single quote for all
> date values; (X)OR remove _all_ quotes, and change the separator ':'
> to 'M', 'W', 'D' according to the data frequency, in analogy
to the
> 'Q' for quarterly data. What do you think?
Sounds reasonable. One consideration is the possibility of
"round-tripping". That is, ideally if you export gretl data to CSV then
read them back into gretl, the time-series information should be
preserved. I'm not sure if that's the case at present, but it's
something I've thought of (at least intermittently) when messing with
the exporter/importer code.
Well for monthly and weekly I think replacing the ':' by 'M' and
'W'
would help here. For higher frequencies it's probably more complicated.
Anyway, I would welcome it if gretl's csv export could become a little
more consistent in the future.
Btw, the background for all this is that I want to manage data in gretl
and then pass them to specialized python programs via temporary csv
files that are created in gretl scripts; these scripts then invoke the
python programs via the shell ('!' command). Actually I would like to
wrap those things in a fancy function package, but I found out that one
cannot use '!' in a function apparently. (Same for 'include'.) Could
that restriction be lifted?
(And if you wonder why I'm not simply doing everything in a gretl
script, the main reason is lack of object-orientation there.)
<snip>
One possibility I've had in mind for a while is merging
everything under
"open" (scrap the "import" command and associated menu items).
Unfortunately, other issues (such as implementing Arellano-Bond or
fixing the behaviour of sub-sampling commands within user-defined
functions) have got in the way ;-)
of course it's not high priority, but it seems like a good idea to me.
Thanks for listening,
Sven
8:22 p.m.
On Sat, 16 Dec 2006, Sven Schreiber wrote:
Well if 'store' is expert-ish, then it needs some expert
information ;-) Anyway, I would prefer if 'store' simply uses
the gretl preference setting of the user directory. At least
that is what I expected first.
OK, I agree that's not an unreasonable expectation. I'll modify
"store" to do that, but leave the option of using a leading dot to
mean "save in the PWD", as in
store ./foo.csv --csv
>> All this makes parsing gretl's csv files error-prone
IMHO. I
>> propose to do one of the following two things: Either enclose
>> _all_ strings in double quotes, and have the
>> enclosed-but-leading single quote for all date values; (X)OR
>> remove _all_ quotes, and change the separator ':' to 'M',
'W',
>> 'D' according to the data frequency, in analogy to the 'Q' for
>> quarterly data. What do you think?
I'm inclined to remove all quotes, since none of the fields will
have embedded commas, or embedded spaces.
What do you think of the idea, for quarterly and monthly data, of
putting in the "date" or "obs" field the date which opens the
period, in YYYY-MM-DD notation (or YYYY/MM/DD??). This would mean
that a spreadsheet program ought to be able to get the time-series
information right. For annual we'd just print the year.
Btw, the background for all this is that I want to manage data
in gretl and then pass them to specialized python programs...
Ah, I'm all in favour of computer-friendliness! I'll try to
cooperate.
Allin.
9:12 p.m.
Allin Cottrell schrieb:
On Sat, 16 Dec 2006, Sven Schreiber wrote:
OK, I agree that's not an unreasonable expectation. I'll modify
"store"
to do that, but leave the option of using a leading dot to mean "save in
the PWD", as in
store ./foo.csv --csv
thanks, that's great!
I'm inclined to remove all quotes, since none of the fields will have
embedded commas, or embedded spaces.
sounds good
What do you think of the idea, for quarterly and monthly data, of
putting in the "date" or "obs" field the date which opens the period,
in
YYYY-MM-DD notation (or YYYY/MM/DD??). This would mean that a
spreadsheet program ought to be able to get the time-series information
right. For annual we'd just print the year.
Well, the disadvantage is that a parser could not infer the frequency
from one date alone. (Apart from the special-casing issue for annual
data.) So I would rather keep something like yyyyq and yyyymm; in
principle we don't need any separator at all, the rule "4 digits:
annual, 5 digits: quarterly, 6 digits: monthly: 8 digits: daily" would
be enough. However, I can also see the virtue of being a little more
explicit, so some separator characters are fine with me.
(Oh, I almost forgot about weekly data, but I guess the gretl's current
practice of using starting days' dates is the best one can do.)
> Btw, the background for all this is that I want to manage data in
> gretl and then pass them to specialized python programs...
Ah, I'm all in favour of computer-friendliness! I'll try to cooperate.
May I repeat the question about the (im)possibility of shell escapes
('!') in functions? Then one could call any external engine one likes
and provide a nice gretl function package interface for others (and
oneself). For example it may also be nice to plug in some R routines in
this way.
good night,
sven
3 a.m.
On Sun, 17 Dec 2006, Sven Schreiber wrote:
> OK, I agree that's not an unreasonable expectation. I'll
> modify "store" to do that, but leave the option of using a
> leading dot to mean "save in the PWD", as in
>
> store ./foo.csv --csv
thanks, that's great!
Done, in CVS.
> What do you think of the idea, for quarterly and monthly data,
> of putting in the "date" or "obs" field the date which opens
> the period, in YYYY-MM-DD notation (or YYYY/MM/DD??). This
> would mean that a spreadsheet program ought to be able to get
> the time-series information right. For annual we'd just print
> the year.
Well, the disadvantage is that a parser could not infer the
frequency from one date alone. (Apart from the special-casing
issue for annual data.) So I would rather keep something like
yyyyq and yyyymm; in principle we don't need any separator at
all, the rule "4 digits: annual, 5 digits: quarterly, 6 digits:
monthly: 8 digits: daily" would be enough. However, I can also
see the virtue of being a little more explicit, so some
separator characters are fine with me.
For now, in CVS I have, for example, "1975Q3" for quarterly, and
"2004M07" for monthly (without the surrounding quotes in the
actual CSV file). I have made sure that CVS gretl can read these
formulations back from CSV and get the dates right.
(Oh, I almost forgot about weekly data, but I guess the gretl's
current
practice of using starting days' dates is the best one can do.)
Yes.
May I repeat the question about the (im)possibility of shell
escapes ('!') in functions? Then one could call any external
engine one likes and provide a nice gretl function package
interface for others (and oneself). For example it may also be
nice to plug in some R routines in this way.
I see your point. My concern is to ensure that gretl doesn't
become a vector for malware. Maybe this is paranoid, but paranoia
seems reasonable given the current state of the internet. Anyone
else have thoughts on this?
Allin.
8:32 a.m.
On Mon, December 18, 2006 04:00, Allin Cottrell wrote:
> May I repeat the question about the (im)possibility of shell
> escapes ('!') in functions? Then one could call any external
> engine one likes and provide a nice gretl function package
> interface for others (and oneself). For example it may also be
> nice to plug in some R routines in this way.
I see your point. My concern is to ensure that gretl doesn't
become a vector for malware. Maybe this is paranoid, but paranoia
seems reasonable given the current state of the internet. Anyone
else have thoughts on this?
I agree with Sven. Being multi-platform is a big asset here. Even if someone
may conceivably inbed a line like
!sudo rm -rf /
in a script, I'm sure windows users will notice that the script isn't working
for some obscure reason and the above cunning stratagem will be rather
short-lived.
Riccardo (Jack) Lucchetti
Dipartimento di Economia
Facoltà di Economia "G. Fuà"
Ancona
10:45 a.m.
Riccardo (Jack) Lucchetti schrieb:
On Mon, December 18, 2006 04:00, Allin Cottrell wrote:
>> May I repeat the question about the (im)possibility of shell
>> escapes ('!') in functions? Then one could call any external
>> engine one likes and provide a nice gretl function package
>> interface for others (and oneself). For example it may also be
>> nice to plug in some R routines in this way.
> I see your point. My concern is to ensure that gretl doesn't
> become a vector for malware. Maybe this is paranoid, but paranoia
> seems reasonable given the current state of the internet. Anyone
> else have thoughts on this?
I agree with Sven. Being multi-platform is a big asset here. Even if someone
may conceivably inbed a line like
!sudo rm -rf /
in a script, I'm sure windows users will notice that the script isn't working
for some obscure reason and the above cunning stratagem will be rather
short-lived.
Thanks Jack, although I'm not completely sure that your optimism is
warranted...
Two thoughts: First, gui-gretl's design is good in that the user needs
to specifically allow shell commands in the preferences. Also, for
function packages one could tell authors that the function needs to
print a warning at the end if it made use of shell commands. Something
like: "Don't forget to disable shell commands again in your preferences
for security reasons." Of course, one would also have to think of how to
secure the function package repository against unwanted alterations.
(What's the current state there?) But in the end I think it's reasonably
safe.
Secondly, Allin's point made me think about command-line gretl. I mean
shell commands are allowed in normal scripts, so what security exists
when a script is executed? Is there a switch like the one in the
gui-preferences?
thanks,
sven
4:50 p.m.
On Mon, 18 Dec 2006, Sven Schreiber wrote:
Two thoughts: First, gui-gretl's design is good in that the user
needs to specifically allow shell commands in the preferences.
Yes -- in light of that, maybe it's OK to enable shell commands in
functions.
Secondly, Allin's point made me think about command-line gretl.
I mean shell commands are allowed in normal scripts, so what
security exists when a script is executed? Is there a switch
like the one in the gui-preferences?
Shell commands are disabled by default in gretlcli, but they are
enabled in gretlcli if they're enabled via the GUI Preferences
(and then disabled again if they're turned off in the GUI).
There's no separate switch.
Allin.
10:22 a.m.
Allin Cottrell schrieb:
On Sun, 17 Dec 2006, Sven Schreiber wrote:
>> OK, I agree that's not an unreasonable expectation. I'll modify
>> "store" to do that, but leave the option of using a leading dot to
>> mean "save in the PWD", as in
>>
>> store ./foo.csv --csv
>
> thanks, that's great!
Done, in CVS.
This is not working for me, unfortunately. (I'm reasonably sure I have a
recent enough CVS build because ! escapes now work in functions...) The
files are still in my standard home-dir which is different from my gretl
setting.
Btw, I noticed the save/open tab in preferences, where you can choose
between gretl user dir and current working dir. Does that setting affect
"store"?
thanks,
sven
4:25 p.m.
On Fri, 22 Dec 2006, Sven Schreiber wrote:
>>> OK, I agree that's not an unreasonable expectation.
I'll modify
>>> "store" to do that, but leave the option of using a leading dot to
>>> mean "save in the PWD", as in
>>>
>>> store ./foo.csv --csv
>>
>> thanks, that's great!
>
> Done, in CVS.
This is not working for me, unfortunately. (I'm reasonably sure I have a
recent enough CVS build because ! escapes now work in functions...) The
files are still in my standard home-dir which is different from my gretl
setting.
Btw, I noticed the save/open tab in preferences, where you can choose
between gretl user dir and current working dir. Does that setting affect
"store"?
Till now it has not affected the behaviour of "store", but since
it seems as if it should, I've made that modification now.
In what context are you calling "store"? I've done a fair amount
of testing and find that it uses the gretl user dir unless you
give an absolute path, or start the filename with "." (and now,
unless you have set a preference for using the current directory).
Allin.
4:31 p.m.
On Fri, 22 Dec 2006, Allin Cottrell wrote:
> Btw, I noticed the save/open tab in preferences, where you can
choose
> between gretl user dir and current working dir. Does that setting affect
> "store"?
Till now it has not affected the behaviour of "store", but since
it seems as if it should, I've made that modification now.
To allow this sort of thing to be configured via gretlcli, I've
also added a relevant "set" parameter
set use_cwd 1 # store will use the current dir
set use_cwd 0 # store uses the gretl user dir
The default is 0.
Allin.
10:19 a.m.
Allin Cottrell schrieb:
In what context are you calling "store"? I've done a fair amount of
testing and find that it uses the gretl user dir unless you give an
absolute path, or start the filename with "." (and now, unless you have
set a preference for using the current directory).
I suspect it's an install issue on my side, because I also still see
quotes in the exported csv files. Maybe some leftovers from the 1.6.0
release version, although I tried to delete all of them. Strange that I
still have the new thing that "!" works in functions, though.
I will try to fix it and report back. (Again, any hints on how to remove
all of a self-compiled gretl from a ubuntu or debian system?)
happy holidays,
sven
5:48 p.m.
On Sun, 24 Dec 2006, Sven Schreiber wrote:
I will try to fix it and report back. (Again, any hints on how to
remove
all of a self-compiled gretl from a ubuntu or debian system?)
My habit is to configure and install into a special location, e.g.
/opt/gretl. Then blowing it all away is simply a matter of "rm
-rf /opt/gretl". To get this working properly you need to add,
say, opt/gretl/bin to your path, and add /opt/gretl/lib to
/etc/ld.so.conf (or use LD_LIBRARY_PATH, maybe).
Allin.
11:32 p.m.
Sven Schreiber schrieb:
I suspect it's an install issue on my side, because I also still see
quotes in the exported csv files. Maybe some leftovers from the 1.6.0
release version, although I tried to delete all of them. Strange that I
still have the new thing that "!" works in functions, though.
Ok it's working now, I guess the checkinstall of the previous
cvs-version did not work properly after all. Now did a standard make
install instead.
Thanks for fulfilling all these cvs-related wishes!
-sven
6330
days inactive
6342
days old
16 comments
3 participants
participants (3)
-
Allin Cottrell -
Riccardo (Jack) Lucchetti -
Sven Schreiber