6:10 p.m.
Hi,
well, not Alfred himself, but "his" data -- I'm talking about the
real-time database in St. Louis, of course.
I'm working on some functions which would allow to access the Alfred
data directly from the (tab-delimited text) files that you can download
there, and of course I'm running into problems (which is not
unexpected). For example, consider the following attempt to get the
latest vintage/publication of each datapoint from the attached partial
file:
<hansl>
nulldata 500 --preserve
setobs 12 1960:01 --time-series # monthly =12 in this example
join INDPRO_excerpt.txt INDPRO --tkey=observation_date
--filter="realtime_end_date==."
</hansl>
There are at least two issues here already:
1) I had to transform the "observation_date" column to match the monthly
frequency in the gretl workfile: Alfred uses the first day of the month
to indicate that month, like "1980-02-01" for February 1980. So I
chopped off the trailing "-01" part of that column to get
gretl-compatible monthly date strings.
In the future it would be nice if the join command could do that
automatically/internally for the column indicated by the "--tkey"
option, in order to support Alfred.
(For completeness, for quarterly data it's the same, Alfred writes e.g.
"1980-04-01" for 1980Q2.)
2) The above command still produces an error, namely:
"join: missing string in filtering"
I thought it has to do with the dot "." (which indicates NA in Alfred's
text file output, with Alfred's XLS files it's "#NV"), perhaps it
wasn't
recognized by gretl/join as a string constant. So I replaced these dots
with explicit "na" in the text file, changed the --filter option
accordingly, but still the same error.
This is where I could use some ideas, really! -- Note that the
"realtime_end_date" column also holds daily date strings apart from the
dots, so maybe gretl gets confused there?
Thanks,
Sven
7:14 p.m.
On Mon, 19 Aug 2013, Sven Schreiber wrote:
well, not Alfred himself, but "his" data -- I'm talking
about the real-time
database in St. Louis, of course.
I'm working on some functions which would allow to access the Alfred data
directly from the (tab-delimited text) files that you can download there, and
of course I'm running into problems (which is not unexpected). For example,
consider the following attempt to get the latest vintage/publication of each
datapoint from the attached partial file:
<hansl>
nulldata 500 --preserve
setobs 12 1960:01 --time-series # monthly =12 in this example
join INDPRO_excerpt.txt INDPRO --tkey=observation_date
--filter="realtime_end_date==."
</hansl>
There are at least two issues here already:
1) I had to transform the "observation_date" column to match the monthly
frequency in the gretl workfile: Alfred uses the first day of the month to
indicate that month, like "1980-02-01" for February 1980. So I chopped off
the trailing "-01" part of that column to get gretl-compatible monthly date
strings.
No need for that. Use the option --time="%Y-%m-%d" to set the appropriate
time format. (See the current CVS User's Guide.)
2) The above command still produces an error, namely:
"join: missing string in filtering"
I thought it has to do with the dot "." (which indicates NA in Alfred's
text
file output, with Alfred's XLS files it's "#NV"), perhaps it wasn't
recognized by gretl/join as a string constant.
Ah, there's an interesting "gotcha" here. The field "." in
delimited-text
data input is mapped to NA by gretl (as in fact intended by Alfred), so
such observations of realtime_end_date are treated as missing, not as the
string literal ".". Hence the error message about a "missing string".
One
way around that problem would be to introduce a further filtering option
such as
--filter=missing(right-hand-colname)
In the meantime one can replace "." in the input by a dummy string that
doesn't map to NA for gretl, e.g. "x" then do
--filter="realtime_end_date==x"
(But, agreed, it's not nice to have to pre-process the input.)
Allin
8:23 p.m.
Thanks for the prompt reply!
Am 19.08.2013 21:14, schrieb Allin Cottrell:
On Mon, 19 Aug 2013, Sven Schreiber wrote:
>
> 1) I had to transform the "observation_date" column to match the monthly
> frequency in the gretl workfile: Alfred uses the first day of the month to
> indicate that month, like "1980-02-01" for February 1980. So I chopped off
> the trailing "-01" part of that column to get gretl-compatible monthly
date
> strings.
No need for that. Use the option --time="%Y-%m-%d" to set the appropriate
time format. (See the current CVS User's Guide.)
Ah ok -- I knew that option to specify the input format, but I didn't
know that gretl would then accept the recognized daily dates as monthly
date labels. Very good, it works. Would it even work with quarterly
frequency? (I will find out myself eventually....)
> 2) The above command still produces an error, namely:
> "join: missing string in filtering"
> I thought it has to do with the dot "." (which indicates NA in Alfred's
text
> file output, with Alfred's XLS files it's "#NV"), perhaps it
wasn't
> recognized by gretl/join as a string constant.
Ah, there's an interesting "gotcha" here. The field "." in
delimited-text
data input is mapped to NA by gretl (as in fact intended by Alfred), so
such observations of realtime_end_date are treated as missing, not as the
string literal ".". Hence the error message about a "missing string".
One
way around that problem would be to introduce a further filtering option
such as
I'm wondering whether gretl isn't perhaps doing "too much too early"
with the input file contents in the context of the join command. After
all, the values that are going to go into the gretl workfile are only in
the 'varname' column. The other columns are "only" used for filtering
and matching, so maybe they should be left as is? (With the further
exception of the column specified by --tkey, I guess)
--filter=missing(right-hand-colname)
Yes perhaps.
In the meantime one can replace "." in the input by a dummy string that
doesn't map to NA for gretl, e.g. "x" then do
--filter="realtime_end_date==x"
(But, agreed, it's not nice to have to pre-process the input.)
It works for now -- thanks!
Cheers,
sven
9:34 p.m.
On Mon, 19 Aug 2013, Sven Schreiber wrote:
Thanks for the prompt reply!
Am 19.08.2013 21:14, schrieb Allin Cottrell:
> On Mon, 19 Aug 2013, Sven Schreiber wrote:
>>
>> 1) I had to transform the "observation_date" column to match the
monthly
>> frequency in the gretl workfile: Alfred uses the first day of the month to
>> indicate that month, like "1980-02-01" for February 1980. So I chopped
off
>> the trailing "-01" part of that column to get gretl-compatible monthly
date
>> strings.
>
> No need for that. Use the option --time="%Y-%m-%d" to set the appropriate
> time format. (See the current CVS User's Guide.)
Ah ok -- I knew that option to specify the input format, but I didn't
know that gretl would then accept the recognized daily dates as monthly
date labels. Very good, it works. Would it even work with quarterly
frequency? (I will find out myself eventually....)
Yes, it should work for quarterly data too.
>> 2) The above command still produces an error, namely:
>> "join: missing string in filtering"
>> I thought it has to do with the dot "." (which indicates NA in
Alfred's text
>> file output, with Alfred's XLS files it's "#NV"), perhaps it
wasn't
>> recognized by gretl/join as a string constant.
>
> Ah, there's an interesting "gotcha" here. The field "." in
delimited-text
> data input is mapped to NA by gretl (as in fact intended by Alfred), so
> such observations of realtime_end_date are treated as missing, not as the
> string literal ".". Hence the error message about a "missing
string". One
> way around that problem would be to introduce a further filtering option
> such as
I'm wondering whether gretl isn't perhaps doing "too much too early"
with the input file contents in the context of the join command. After
all, the values that are going to go into the gretl workfile are only in
the 'varname' column. The other columns are "only" used for filtering
and matching, so maybe they should be left as is? (With the further
exception of the column specified by --tkey, I guess)
I think that in most cases we have to try to recognize missing values in
the outer data file. But this can be tricky with string variables, after
all "." could be intended as a valid value for a string variable (as could
"NA" for that matter). In the Alfred case I'd say that gretl got it right
but we might not always be so lucky.
> --filter=missing(right-hand-colname)
Yes perhaps.
That's now in CVS on an experimental basis. You can also filter on
!missing(right-hand-colname).
Allin
10:49 p.m.
On Mon, 19 Aug 2013, Allin Cottrell wrote:
On Mon, 19 Aug 2013, Sven Schreiber wrote:
>
> I'm wondering whether gretl isn't perhaps doing "too much too
early"
> with the input file contents in the context of the join command. After
> all, the values that are going to go into the gretl workfile are only in
> the 'varname' column. The other columns are "only" used for
filtering
> and matching, so maybe they should be left as is? (With the further
> exception of the column specified by --tkey, I guess)
I think that in most cases we have to try to recognize missing values in the
outer data file. But this can be tricky with string variables, after all "."
could be intended as a valid value for a string variable (as could "NA" for
that matter).
Thinking about this some more, it seems we really should not apply
the same policy regarding CSV NAs to string variables as we apply to
numeric ones. With numeric variables we want to be fairly "liberal",
interpreting as NA any string that the author of the CSV file could
plausibly have intended as a code for missing. But there's no
unambiguous "missing" indiactor for string values, other than a
blank cell or an empty string.
So here's a suggestion: when we determine that a certain column of a
CSV file represents a string-valued variable, by default we treat
all non-blank values as string literals. But we provide a "set"
variable ("missing_string" or some such) so that the user can
specify a missing-code for string-valued input. E.g. when reading
from Alfred's tab-separated files one could say
set missing_string "."
(This would not just be for "join", but for any delimited-text
read.)
I'll be mostly offline for a week or so, so I won't be able to
implement this right away. But I'll be interested to hear what
people think.
Allin
10:58 p.m.
On Mon, 19 Aug 2013, Allin Cottrell wrote:
On Mon, 19 Aug 2013, Allin Cottrell wrote:
So here's a suggestion: when we determine that a certain column of a
CSV file represents a string-valued variable, by default we treat
all non-blank values as string literals. But we provide a "set"
variable ("missing_string" or some such) so that the user can
specify a missing-code for string-valued input. E.g. when reading
from Alfred's tab-separated files one could say
set missing_string "."
(This would not just be for "join", but for any delimited-text
read.)
We already have "set csv_na"; we could use that.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
12:42 a.m.
On Tue, 20 Aug 2013, Riccardo (Jack) Lucchetti wrote:
On Mon, 19 Aug 2013, Allin Cottrell wrote:
> On Mon, 19 Aug 2013, Allin Cottrell wrote:
>
> So here's a suggestion: when we determine that a certain column of a
> CSV file represents a string-valued variable, by default we treat
> all non-blank values as string literals. But we provide a "set"
> variable ("missing_string" or some such) so that the user can
> specify a missing-code for string-valued input. E.g. when reading
> from Alfred's tab-separated files one could say
>
> set missing_string "."
>
> (This would not just be for "join", but for any delimited-text
> read.)
We already have "set csv_na"; we could use that.
That occurred to me too, but I think it would be confusing. The
(only) role of "set csv_na" at present is to set the string used to
represent NA on _output_ of CSV from gretl. This string may be
specific to the anticipated use of gretl's CSV output by a
third-party program (e.g. Stata or Ox, which have their own notions
of what represents NA).
But what we're talking about now is the string to interpret as NA on
_input_ of string-valued variables from CSV. This string may be
specific to the third-party producer of the CSV file (e.g. Alfred).
The string that's appropriate for one of these uses is not
necessarily appropriate for the other use; or to put it differently,
"set csv_na" (used as per its current role) could have unwanted
side-effects if it were also to govern the reading of string-valued
variables from CSV subsequently.
Allin
7:40 a.m.
Am 20.08.2013 02:42, schrieb Allin Cottrell:
On Tue, 20 Aug 2013, Riccardo (Jack) Lucchetti wrote:
> On Mon, 19 Aug 2013, Allin Cottrell wrote:
>
>> On Mon, 19 Aug 2013, Allin Cottrell wrote:
>>
>> So here's a suggestion: when we determine that a certain column of a
>> CSV file represents a string-valued variable, by default we treat
>> all non-blank values as string literals. But we provide a "set"
>> variable ("missing_string" or some such) so that the user can
>> specify a missing-code for string-valued input. E.g. when reading
>> from Alfred's tab-separated files one could say
>>
>> set missing_string "."
>>
>> (This would not just be for "join", but for any delimited-text
>> read.)
>
> We already have "set csv_na"; we could use that.
That occurred to me too, but I think it would be confusing. The
(only) role of "set csv_na" at present is to set the string used to
represent NA on _output_ of CSV from gretl. This string may be
specific to the anticipated use of gretl's CSV output by a
third-party program (e.g. Stata or Ox, which have their own notions
of what represents NA).
But what we're talking about now is the string to interpret as NA on
_input_ of string-valued variables from CSV. This string may be
specific to the third-party producer of the CSV file (e.g. Alfred).
The string that's appropriate for one of these uses is not
necessarily appropriate for the other use; or to put it differently,
"set csv_na" (used as per its current role) could have unwanted
side-effects if it were also to govern the reading of string-valued
variables from CSV subsequently.
I agree; actually I had looked up the docs for that 'set' option because
it sounded like it could be relevant. There may be a case to rename (or
introduce aliases for) these options to distinguish input and output
more clearly, like "set csv_write_na" and "set csv_read_na". But then
you knew already that I'm always for more verbosity and explicitness...
thanks,
sven
7:15 a.m.
Am 19.08.2013 23:34, schrieb Allin Cottrell:
On Mon, 19 Aug 2013, Sven Schreiber wrote:
>> --filter=missing(right-hand-colname)
>
> Yes perhaps.
That's now in CVS on an experimental basis. You can also filter on
!missing(right-hand-colname).
Just a tiny remark; since we have ok() as equivalent for !missing(), I
think it should be allowed here, too.
-s
7:40 a.m.
Am 19.08.2013 21:14, schrieb Allin Cottrell:
On Mon, 19 Aug 2013, Sven Schreiber wrote:
> well, not Alfred himself, but "his" data -- I'm talking about the
real-time
> database in St. Louis, of course.
>
...
(But, agreed, it's not nice to have to pre-process the input.)
Status update: For now it still seems necessary to pre-process the
input. I did that with Python in a 'foreign' block within the gretl
script, and after some trial-and-error passing around the correctly
formatted path strings it works for me as a temporary solution. I will
send the code when I have wrapped the stuff in functions as far as possible.
Here's how I transformed the Alfred input file (of course I may have
overlooked some nicer solutions):
As you know, the publication date columns (realtime_start_date and
realtime_end_date) come in daily ISO format %Y-%m-%d. But I need to do
an earlier/later filtering operation on them. So I stripped the hyphens
to get pure numbers.
(BTW: I tried to do the same thing with the --tkey column
"observation_date" and specify the format there with
'--time="%Y%m%d"'
[and also "%Y %m %d"], but that produced an error. Expected or a bug?)
I also changed the missing value code in the realtime_end_date from "."
to the number 99999999 for the same reason, to make numerical comparison
possible.
In order to avoid this preprocessing, it would be great if it were
possible to use a column with daily ISO dates in a filtering comparison
directly, as in:
'--filter="realtime_start_date<=20130820"' (or ... <=2013-08-20)
Maybe accompanied with a new option to declare which columns should be
treated as holding ISO-date data:
'--isodates=realtime_start_date,realtime_end_date'
And finally, a wishlist for 'join' options. These missing features can
be worked around, but it would be easier if they existed:
1) '--filter' with two conditions instead of just one; in the realtime
context the natural application is to do a "within" check:
realtime_start_date <= somedate <= realtime_end_date
2) '--aggr="seq:last"', to get the last matching item, where the number
of that last item can vary between different key-matched sequences.
Thanks,
Sven
11:07 a.m.
On Wed, 21 Aug 2013, Sven Schreiber wrote:
Am 19.08.2013 21:14, schrieb Allin Cottrell:
> On Mon, 19 Aug 2013, Sven Schreiber wrote:
>
>> well, not Alfred himself, but "his" data -- I'm talking about the
real-time
>> database in St. Louis, of course.
>>
...
>
> (But, agreed, it's not nice to have to pre-process the input.)
>
Status update: For now it still seems necessary to pre-process the
input. I did that with Python in a 'foreign' block within the gretl
script, and after some trial-and-error passing around the correctly
formatted path strings it works for me as a temporary solution. I will
send the code when I have wrapped the stuff in functions as far as possible.
Here's how I transformed the Alfred input file (of course I may have
overlooked some nicer solutions):
[...]
Alternative approach: in the attached tarball you'll find
1) an unmodified file from Alfred (if you choose the "Observations by
Real-Time Period" format, like Sven did)
2) a hansl script (alfred_test.inp), which works on the file above, and
performs all the required preprocessing without having to resort to
Python. This file will give an error at the "join" command, which is
the same error that Sven got originally; note that, by using the
--tkey and --time options, the outer key field is ok "as-it-comes".
3) a diff file which, if applied to CVS, will make the source of the error
evident: in practice, the string->codes transformation of the string
variable used for filtering is not done properly.
I'd like to point out that (3) is a bug we'll have to fix anyway, whatever
solution we agree upon in order to code missing values in the outer file.
Unfortunately, I won't be able to work on this today or tomorrow.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
6:05 p.m.
On Wed, 21 Aug 2013, Riccardo (Jack) Lucchetti wrote:
Alternative approach: in the attached tarball you'll find
1) an unmodified file from Alfred (if you choose the "Observations by
Real-Time Period" format, like Sven did)
2) a hansl script (alfred_test.inp), which works on the file above, and
performs all the required preprocessing without having to resort to
Python. This file will give an error at the "join" command, which is
the same error that Sven got originally; note that, by using the
--tkey and --time options, the outer key field is ok "as-it-comes".
3) a diff file which, if applied to CVS, will make the source of the error
evident: in practice, the string->codes transformation of the string
variable used for filtering is not done properly.
I'd like to point out that (3) is a bug we'll have to fix anyway, whatever
solution we agree upon in order to code missing values in the outer file.
Unfortunately, I won't be able to work on this today or tomorrow.
Hmm, I think you just forgot to use the nice file you created with
the dots replaced, in alfred_test.inp. If you stick in the line
fname = "(a)dotdir/tmp.csv"
before invoking "join" it works fine.
Allin
7:03 p.m.
On Thu, 22 Aug 2013, Allin Cottrell wrote:
Hmm, I think you just forgot to use the nice file you created with
the dots
replaced, in alfred_test.inp. If you stick in the line
fname = "(a)dotdir/tmp.csv"
before invoking "join" it works fine.
*blushes*
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
7:40 p.m.
On Thu, 22 Aug 2013, Riccardo (Jack) Lucchetti wrote:
On Thu, 22 Aug 2013, Allin Cottrell wrote:
> Hmm, I think you just forgot to use the nice file you created with the dots
> replaced, in alfred_test.inp. If you stick in the line
>
> fname = "(a)dotdir/tmp.csv"
>
> before invoking "join" it works fine.
*blushes*
Well, my turn to blush. I realize I forgot to commit the
modification I mentioned recently, allowing missing() as a filtering
option for join. And the machine on which the new code resides is
a couple of thousand miles away (I'm now in the Wild West). I'll do
that next week.
Allin
7:43 p.m.
Am 21.08.2013 09:40, schrieb Sven Schreiber:
Status update: For now it still seems necessary to pre-process the
input. I did that with Python in a 'foreign' block within the gretl
script, and after some trial-and-error passing around the correctly
formatted path strings it works for me as a temporary solution. I will
send the code when I have wrapped the stuff in functions as far as possible.
I'm attaching a hansl file with a couple of relevant functions which
should cover many real-world cases. These functions assume an Alfred csv
file preprocessed like I described before, which resides in the location
specified in the string argument 'fname'. (This path is taken as-is, so
the caller is responsible for getting it right.)
These are the function signatures, where 'vname' is the string variable
holding the variable name as in the Alfred file:
1) getAlf_ithPub(fname,vname,i[0::1])
Retrieves for each observation period (= what I have also called
reference period sometimes) the i-th published value, no matter when
that publication occurred. Can lead to NAs, e.g. for the future or for
high publication numbers.
The case i=0 is special, it gets the final publication for each obs period.
Examples:
series @vname_latest = getAlf_ithPub(fname,vname,0)
series @vname_second = getAlf_ithPub(fname,vname,2)
2) getAlfVintage(fname,vname,day)
Retrieves the vintage valid on date 'day' (which is a string argument in
ISO format), for all obs periods. This is perhaps the classic meaning of
realtime data series. If the vintage date is the present, then this
should coincide with the latest publication: getAlf_ithPub(fname,vname,0).
Will also lead to NAs, e.g. for obs periods that are younger than the
vintage date.
Example:
series @vname_19830302 = getAlfVintage(fname,vname,"1983-03-02")
3) getAlfPubafterN(fname,vname,n)
Retrieves for each obs period the publication which was current as of
(the first day of) the n-th period (quarter or month) after the
respective obs period. So in some sense it imposes a constant
publication lag of n periods, no matter how many publications occur in
between. Also possible in principle is n=0, for variables where the
first publication occurse instantaneously (financial markets?). OTOH,
can lead to NAs especially for low n.
Example:
series @vname_after2 = getAlfPubafterN(fname,vname,2)
In all these cases, the dating (=time axis) in the gretl workfile is
according to observation or reference periods. The publication time of
each value differs in general and depends on the context in a relatively
complicated way.
Of course, much more testing needed.
cheers,
sven
7:06 a.m.
On Wed, 21 Aug 2013, Sven Schreiber wrote:
Am 21.08.2013 09:40, schrieb Sven Schreiber:
> Status update: For now it still seems necessary to pre-process the
> input. I did that with Python in a 'foreign' block within the gretl
> script, and after some trial-and-error passing around the correctly
> formatted path strings it works for me as a temporary solution. I will
> send the code when I have wrapped the stuff in functions as far as
> possible.
>
I'm attaching a hansl file with a couple of relevant functions which should
cover many real-world cases. These functions assume an Alfred csv file
preprocessed like I described before, which resides in the location specified
in the string argument 'fname'. (This path is taken as-is, so the caller is
responsible for getting it right.)
Very nice. I've been working on a similar thing myself: I've been trying
to re-implement some of the stuff Yi-Nung has been working on
(downloading financial data from Yahoo) in hansl, without having to go
through R to download the data and pre-process it.
Could you share the details of the pre-processing of the Alfred data you
need? It's very possible that this could be done in hansl directly,
thereby removing the python dependency.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
9:21 a.m.
Am 23.08.2013 09:06, schrieb Riccardo (Jack) Lucchetti:
On Wed, 21 Aug 2013, Sven Schreiber wrote:
> Am 21.08.2013 09:40, schrieb Sven Schreiber:
>> Status update: For now it still seems necessary to pre-process the
>> input. I did that with Python in a 'foreign' block within the gretl
>> script, and after some trial-and-error passing around the correctly
>> formatted path strings it works for me as a temporary solution. I will
>> send the code when I have wrapped the stuff in functions as far as
>> possible.
>>
>
> I'm attaching a hansl file with a couple of relevant functions which
> should cover many real-world cases. These functions assume an Alfred
> csv file preprocessed like I described before, which resides in the
> location specified in the string argument 'fname'. (This path is taken
> as-is, so the caller is responsible for getting it right.)
Could you share the details of the pre-processing of the Alfred data
you
need? It's very possible that this could be done in hansl directly,
thereby removing the python dependency.
The transformation is basically two things:
1) For the 'realtime_start_date' and 'realtime_end_date' columns
transform the entries from ISO format (%Y-%m-%d) to a number string
(%Y%m%d), removing the hyphens, for example "1983-02-03" ->
"19830203".
2) Also for these columns, replace dots (".", missing values) by
"99999999".
Note that your alfred_test.inp code is "not clever" enough, because
AFAICS it also replaces dots in the value column (in your case, with
"_latest_"), which is unwanted I think. (In the reduced example file,
this didn't occur, but it can in general.)
Also, it's not as easy as just removing all the hyphens in the file,
that would be easy in hansl. But the 'observation_date' column must stay
in ISO date format, as I wrote in yet another message, since specifying
'--time="%Y%m%d"' didn't work.
Here's my take at doing it in hansl (untested), but let's not forget
that the goal is (IMHO) to make the preprocessing unnecessary
altogether, by enabling 'join' do smaller/greater comparisons on ISO
date strings!
<hansl>
string temp = readfile(fname)
string out = ""
# to use strsplit() we must insert artificial spaces
temp = strsub(temp,"\n","\n ")
i=1
loop while 1 # loop over the lines in file
string line = strsplit(temp,i)
if line=="" # end of file reached
break
endif
# again insert artificial spaces for splitting
line = strsub(line,"\t","\t ")
## do the replacements
# realtime_start_date is the 3rd col
string trans3 = strsub(strsplit(line,3),"-","")
trans3 = strsub(trans3,".","_latest_") # or "99999999"
# and realtime_end_date the 4th
string trans4 = strsub(strsplit(line,4),"-","")
trans4 = strsub(trans4,".","_latest_")
## put the stuff together and add to output
string out += strsplit(line,1) ~ strsplit(line,2) \
~ trans3 ~ trans4
i++
endloop
## write the transformed file
set echo off
outfile @dotdir/temp.csv --write
print out
outfile --close
</hansl>
Attached is my existing python-based preprocessor. The driving code is:
<hansl>
preprocess = 0
##### preprocess the Alfred file with Python: #########
if preprocess
# first write out the meta information
set echo off
outfile @dotdir/temp.py --write # the temp.py name is hardcoded
print "pyfname = r'@fname'" # r to make Python work with backslashes
print "pyoutfname = r'@outfname'"
outfile --close
set echo on
include AlfredPreprocessor.inp # which is Python code in a foreign block
endif
############ end embedded python / preprocessing ######
</hansl>
As I mentioned in another message, wrapping the Python code in a hansl
function currently doesn't work because the indendation is not preserved.
cheers,
sven
1:16 p.m.
On Fri, 23 Aug 2013, Sven Schreiber wrote:
1) For the 'realtime_start_date' and
'realtime_end_date' columns
transform the entries from ISO format (%Y-%m-%d) to a number string
(%Y%m%d), removing the hyphens, for example "1983-02-03" ->
"19830203".
2) Also for these columns, replace dots (".", missing values) by
"99999999".
Note that your alfred_test.inp code is "not clever" enough, because
AFAICS it also replaces dots in the value column (in your case, with
"_latest_"), which is unwanted I think. (In the reduced example file,
this didn't occur, but it can in general.)
Also, it's not as easy as just removing all the hyphens in the file,
that would be easy in hansl. But the 'observation_date' column must stay
in ISO date format, as I wrote in yet another message, since specifying
'--time="%Y%m%d"' didn't work.
Here's my take at doing it in hansl (untested), but let's not forget
that the goal is (IMHO) to make the preprocessing unnecessary
altogether, by enabling 'join' do smaller/greater comparisons on ISO
date strings!
I agree, it was just a matter of curiosity. And strategy, in a way:
because the fundamental question here is: would it be a good idea to
enhance hansl up to the point when you could use it to do things on text
files for which one would use tools like perl, python, awk or a bash
script (with the standard unix tools: grep, sed, join, cut etcetera)?
On one hand, this would be very cool; plus, when you write a script like
yours, dependency on python clearly reduces portability somehow (yeah, I
know...). But on the other hand, one of the strengths of hansl is (IMO)
its being tailored on the needs of the applied economist and not even
trying to be the Swiss army knife. Besides, that's precisely why we
invented the "foreign" block.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
2:02 p.m.
Am 23.08.2013 15:16, schrieb Riccardo (Jack) Lucchetti:
On Fri, 23 Aug 2013, Sven Schreiber wrote:
>
> Here's my take at doing it in hansl (untested), but let's not forget
> that the goal is (IMHO) to make the preprocessing unnecessary
> altogether, by enabling 'join' do smaller/greater comparisons on ISO
> date strings!
I agree, it was just a matter of curiosity. And strategy, in a way:
because the fundamental question here is: would it be a good idea to
enhance hansl up to the point when you could use it to do things on text
files for which one would use tools like perl, python, awk or a bash
script (with the standard unix tools: grep, sed, join, cut etcetera)?
On one hand, this would be very cool; plus, when you write a script like
yours, dependency on python clearly reduces portability somehow (yeah, I
know...). But on the other hand, one of the strengths of hansl is (IMO)
its being tailored on the needs of the applied economist and not even
trying to be the Swiss army knife. Besides, that's precisely why we
invented the "foreign" block.
Exactly -- also, while I'm now used to the string handling in gretl, I
still think it's a bit awkward for a programming language that you can
sometimes write string literals without quotes (think about the smpl
command), and sometimes you need to quote them. So I don't think this
should be stretched too far.
IMHO, hansl is good enough to write these types of code once in a while,
especially if they're reused often and would introduce a dependency
otherwise. I wouldn't want to be constantly doing text file and string
handling in hansl though. (And the python code I posted could probably
have been shrunk quite a bit if I hadn't tried to treat it as a
full-blown csv file, but just had done the low-level string substitions
like in the hansl version.)
One wish item though, about date handling: today I missed an inverse for
the epochday() function, in order to quickly "subtract" a time span from
a date and get the resulting date.
cheers,
sven
2:06 p.m.
Am 23.08.2013 16:02, schrieb Sven Schreiber:
One wish item though, about date handling: today I missed an inverse for
the epochday() function, in order to quickly "subtract" a time span from
a date and get the resulting date.
Sorry, another wish P.S.: And perhaps generalize strsplit() with an
extra optional argument specifying another character (or maybe an n-char
string) that is considered as the splitting mark, instead of a space as
in the default. This seems quite natural, and also easy to implement.
(And backwards-compatible :-)
thanks,
sven
6:15 p.m.
Am 23.08.2013 11:21, schrieb Sven Schreiber:
Here's my take at doing it in hansl (untested), but let's not forget
that the goal is (IMHO) to make the preprocessing unnecessary
altogether, by enabling 'join' do smaller/greater comparisons on ISO
date strings!
Following is now an actually working version, tested with the real-world
1MB file of INDPRO. However, it is very slow, much slower than using my
Python solution it seems to me. Don't know if there are some gretl
string internals that could be sped up.
cheers,
sven
<hansl>
string temp = readfile(fname)
string out = ""
## Need to roughly guess the number of lines in the file
# The min number of chars per line is 3*10+4+6=40
repetitions = ceil(strlen(temp)/40)
string rest = temp
string col1 = ""
string col2 = ""
string col3 = ""
string col4 = ""
set echo off
loop repetitions # loop over the lines in file
sscanf(rest,"%s\t%s\t%s\t%s\n",col1,col2,col3,col4)
string rest = strstr(rest,"\n") + 1 # offset to drop the leading \n
## do the replacements
# realtime_start_date is the 3rd col
string trans3 = strsub(col3,"-","")
trans3 = strsub(trans3,".","99999999") # or "_latest_"
# and realtime_end_date the 4th
string trans4 = strsub(col4,"-","")
trans4 = strsub(trans4,".","99999999")
## put the stuff together and add to output
sprintf temp "@col1\t@col2\t@trans3\t@trans4\n"
string out += temp
if rest=="" # end of file reached
break
endif
endloop
## write the transformed file
set echo off
outfile @outfname --write
print out
outfile --close
</hansl>
12:39 p.m.
Am 24.08.2013 20:15, schrieb Sven Schreiber:
Am 23.08.2013 11:21, schrieb Sven Schreiber:
>
> Here's my take at doing it in hansl (untested), but let's not forget
> that the goal is (IMHO) to make the preprocessing unnecessary
> altogether, by enabling 'join' do smaller/greater comparisons on ISO
> date strings!
>
Following is now an actually working version, tested with the real-world
1MB file of INDPRO. However, it is very slow, much slower than using my
Python solution it seems to me. Don't know if there are some gretl
string internals that could be sped up.
Specifically, the preprocessing including all calling overheads takes
<2sec with the Python solution, and roughly 120sec with native gretl.
I think the crucial lines are the following:
loop repetitions # loop over the lines in file
sscanf(rest,"%s\t%s\t%s\t%s\n",col1,col2,col3,col4)
string rest = strstr(rest,"\n") + 1 # offset to drop the leading \n
That is, there are thousands of operations working on strings holding
(almost) the entire file content (in this case about 1MB as I said). I
have tried to consolidate this into a more clever sscanf line, but that
didn't really help. Glad to take more ideas.
In contrast, in python the file is read line per line. So I don't know
if it's worth it (bearing in mind that actually we don't want any
preprocessing...), but perhaps it would help if the readfile() function
could be extended to automatically (= at the C level) separate the lines
of the file; for example:
<future hansl>
bundle btemp = readfile(fname,1) # new optional 2nd arg to split lines
loop i=1..nelem(btemp) # extended nelem() for number of bundle items
string line = btemp.line$i
... do stuff with line ...
endloop
</future hansl>
Again, not sure if it's worth it.
thanks,
sven
6:08 p.m.
On Sun, 25 Aug 2013, Sven Schreiber wrote:
Am 24.08.2013 20:15, schrieb Sven Schreiber:
> Am 23.08.2013 11:21, schrieb Sven Schreiber:
>
>>
>> Here's my take at doing it in hansl (untested), but let's not forget
>> that the goal is (IMHO) to make the preprocessing unnecessary
>> altogether, by enabling 'join' do smaller/greater comparisons on ISO
>> date strings!
>
> Following is now an actually working version, tested with the real-world
> 1MB file of INDPRO. However, it is very slow, much slower than using my
> Python solution it seems to me. Don't know if there are some gretl
> string internals that could be sped up.
Specifically, the preprocessing including all calling overheads takes
<2sec with the Python solution, and roughly 120sec with native gretl.
I think the crucial lines are the following:
> loop repetitions # loop over the lines in file
> sscanf(rest,"%s\t%s\t%s\t%s\n",col1,col2,col3,col4)
> string rest = strstr(rest,"\n") + 1 # offset to drop the leading \n
>
That is, there are thousands of operations working on strings holding
(almost) the entire file content (in this case about 1MB as I said). I
have tried to consolidate this into a more clever sscanf line, but that
didn't really help. Glad to take more ideas.
In contrast, in python the file is read line per line. [...]
Yes, that will make the difference. I think that to do this sort of
thing efficiently on big files we would need at least one more
function: something like the internal function bufgets(), which
returns the next line of a text buffer held in memory until the
buffer is exhausted (probably with a switch to drop the trailing
newline character).
Greater flexibility would be available if we implemented the suite
of functions fopen, fclose, fgets and friends. This wouldn't be very
difficult since our functions would just be wrappers for the C
library functions, but it does raise the "Swiss army knife" issue as
mentioned by Jack.
Allin
6:49 p.m.
Am 25.08.2013 20:08, schrieb Allin Cottrell:
On Sun, 25 Aug 2013, Sven Schreiber wrote:
>
> That is, there are thousands of operations working on strings holding
> (almost) the entire file content (in this case about 1MB as I said). I
> have tried to consolidate this into a more clever sscanf line, but that
> didn't really help. Glad to take more ideas.
>
> In contrast, in python the file is read line per line. [...]
Yes, that will make the difference. I think that to do this sort of
thing efficiently on big files we would need at least one more
function: something like the internal function bufgets(), which
returns the next line of a text buffer held in memory until the
buffer is exhausted (probably with a switch to drop the trailing
newline character).
This sounds very much like an extended readfile() function to me. (?)
Greater flexibility would be available if we implemented the suite
of functions fopen, fclose, fgets and friends. This wouldn't be very
difficult since our functions would just be wrappers for the C
library functions, but it does raise the "Swiss army knife" issue as
mentioned by Jack.
Yes. Would the former be possible without the latter? If yes, I think it
would be enough.
-sven
4141
days inactive
4147
days old
23 comments
3 participants
participants (3)
-
Allin Cottrell -
Riccardo (Jack) Lucchetti -
Sven Schreiber