1:40 p.m.
Hi,
I will be working with Midas in gretl again soon, and I've just reread
(or let's say, re-browsed) the midas_gretl.pdf guide.
Looking at the section "importation via join", it occurs to me that this
is quite involved. In contrast, "importation from a database" is very
simple.
Shouldn't it be possible to just do something like this:
<hansl-future>
open AWM.gdt # quarterly
join hamilton.gdt EXRITL --compact=spread # monthly to quarterly
</hansl-future>
or perhaps the 'append' command is better suited than join here? So
replace the last line above with (which however gets all series, but
that's OK):
append hamilton.gdt --spread
I guess this append thing would basically be a syntactic shortcut for
this block (more or less):
pd = $pd
store temp.gdt # the currently active dataset
open hamilton.gdt --preserve
dataset compact pd spread
store tempcompact.gdt
open temp.gdt
append tempcompact.gdt
I'm ready to consider working more with the database storage format
(which I almost never have done), but is the above suggestion unreasonable?
thanks,
sven
2:01 p.m.
On Thu, 11 Jan 2018, Sven Schreiber wrote:
Hi,
I will be working with Midas in gretl again soon, and I've just reread (or
let's say, re-browsed) the midas_gretl.pdf guide.
Looking at the section "importation via join", it occurs to me that this is
quite involved. In contrast, "importation from a database" is very simple.
Shouldn't it be possible to just do something like this:
<hansl-future>
open AWM.gdt # quarterly
join hamilton.gdt EXRITL --compact=spread # monthly to quarterly
</hansl-future>
or perhaps the 'append' command is better suited than join here? So replace
the last line above with (which however gets all series, but that's OK):
append hamilton.gdt --spread
I guess this append thing would basically be a syntactic shortcut for this
block (more or less):
pd = $pd
store temp.gdt # the currently active dataset
open hamilton.gdt --preserve
dataset compact pd spread
store tempcompact.gdt
open temp.gdt
append tempcompact.gdt
I'm ready to consider working more with the database storage format
(which I almost never have done), but is the above suggestion
unreasonable?
Not unreasonable. We should be able to engineer a shortcut of this
sort. I'll have to think about whether "append" or "join" is the
better target for a special option.
Allin
3:17 p.m.
On Thu, 11 Jan 2018, Sven Schreiber wrote:
Hi,
I will be working with Midas in gretl again soon, and I've just reread (or
let's say, re-browsed) the midas_gretl.pdf guide.
Looking at the section "importation via join", it occurs to me that this is
quite involved. In contrast, "importation from a database" is very simple.
Shouldn't it be possible to just do something like this:
<hansl-future>
open AWM.gdt # quarterly
join hamilton.gdt EXRITL --compact=spread # monthly to quarterly
</hansl-future>
or perhaps the 'append' command is better suited than join here? So replace
the last line above with (which however gets all series, but that's OK):
append hamilton.gdt --spread
I vote for join. IMO, the join command is kind of a relatively unknown gem
that we have and I think your proposal, if implemented, will make it quite
popular among people who work with time series datasets.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
9:44 p.m.
On Thu, 11 Jan 2018, Riccardo (Jack) Lucchetti wrote:
On Thu, 11 Jan 2018, Sven Schreiber wrote:
> Hi,
>
> I will be working with Midas in gretl again soon, and I've just reread (or
> let's say, re-browsed) the midas_gretl.pdf guide.
> Looking at the section "importation via join", it occurs to me that this is
> quite involved. In contrast, "importation from a database" is very simple.
>
> Shouldn't it be possible to just do something like this:
> <hansl-future>
> open AWM.gdt # quarterly
> join hamilton.gdt EXRITL --compact=spread # monthly to quarterly
> </hansl-future>
>
> or perhaps the 'append' command is better suited than join here? So replace
> the last line above with (which however gets all series, but that's OK):
>
> append hamilton.gdt --spread
I vote for join. IMO, the join command is kind of a relatively unknown gem
that we have and I think your proposal, if implemented, will make it quite
popular among people who work with time series datasets.
I agree that it would be nice to take the opportunity to raise the
profile of "join". But there are two angles here:
1) How do we actually implement this?
2) How do we "package" whatever we implement for users?
On the first point, one option is doing something along the lines
Sven suggested; that is, the internal equivalent of
<hansl>
pd = $pd
store temp.gdt # the current dataset
open hamilton.gdt --preserve # higher-frequency data
dataset compact pd spread
store tempcompact.gdt
open temp.gdt
append tempcompact.gdt
</hansl>
I think this would be relatively straightforward. Internally, we
wouldn't have to store and replace the original data; we can "open"
and compact a dataset such as hamilton.gdt without disturbing the
prior data in userspace.
The other option would be to extend the "join" code itself to handle
compaction on the fly. In a sense this would be cleaner but I'm
pretty sure it would be more complicated; "join" is already quite
complicated.
That leaves the "packaging" question: in principle whatever we come
up with could be presented as as a "join" option even if the
internals are closer to "append".
Allin
12:33 p.m.
Am 11.01.2018 um 22:44 schrieb Allin Cottrell:
On Thu, 11 Jan 2018, Riccardo (Jack) Lucchetti wrote:
> On Thu, 11 Jan 2018, Sven Schreiber wrote:
>> or perhaps the 'append' command is better suited than
join here? So
> I vote for join. IMO, the join command is kind of a relatively unknown
> gem that we have and I think your proposal, if implemented, will make
> it quite popular among people who work with time series datasets.
I agree that it would be nice to take the opportunity to raise the
profile of "join".
1) How do we actually implement this?
The other option would be to extend the "join" code itself
to handle
compaction on the fly. In a sense this would be cleaner but I'm pretty
sure it would be more complicated; "join" is already quite complicated.
Not sure if I understand all the ramifications; in principle I've
thought that the compaction would always come at the very end of what
join does, which would then seem relatively straightforward. EXCEPT:
given that AFAIK join may read inputs line by line instead of
variable-by-variable this of course makes it complex. Is this the reason
for a difficult implementation?
[> 2) How do we "package" whatever we implement for users?]
That leaves the "packaging" question: in principle whatever
we come up
with could be presented as as a "join" option even if the internals are
closer to "append".
I believe that any real solution would also have to cover the real-time
data aspect, because forecasting in real time is going to be an
important application of mixed-frequency data. At first glance this
would suggest join, since there is an entire chapter on it for real-time
data.
OTOH I don't think it would be a problem if a two-stage approach were to
be officially recommended:
1) Do what you need to do with join to get your high-frequency gretl
dataset. This becomes a temporary dataset.
2) Merge this temp data into your low-frequency dataset -- if this stage
is going to be called 'join' or 'append' doesn't matter all that
much.
This would seem to me clean enough to deal with real time data.
my 2c (haven't read that expression in a long time)
sven
8:21 p.m.
On Fri, 12 Jan 2018, Sven Schreiber wrote:
Am 11.01.2018 um 22:44 schrieb Allin Cottrell:
> On Thu, 11 Jan 2018, Riccardo (Jack) Lucchetti wrote:
>> On Thu, 11 Jan 2018, Sven Schreiber wrote:
>>> or perhaps the 'append' command is better suited than join here? So
>
>> I vote for join. IMO, the join command is kind of a relatively unknown gem
>> that we have and I think your proposal, if implemented, will make it quite
>> popular among people who work with time series datasets.
>
> I agree that it would be nice to take the opportunity to raise the profile
> of "join".
> 1) How do we actually implement this?
> The other option would be to extend the "join" code itself to
> handle compaction on the fly. In a sense this would be cleaner but
> I'm pretty sure it would be more complicated; "join" is already
> quite complicated.
Not sure if I understand all the ramifications; in principle I've
thought that the compaction would always come at the very end of
what join does, which would then seem relatively straightforward.
EXCEPT: given that AFAIK join may read inputs line by line instead
of variable-by-variable this of course makes it complex. Is this the
reason for a difficult implementation?
The difficulty I perceived is this: join is set up nicely to do
matching by time on left and right, but in its basic form this assumes
that the time-series frequency is the same on both sides. From that
point of view it would be "too late" to do compaction as the last
step; it would be easier to do compaction first, then match. And that
works fine, as in:
<hansl>
open hamilton.gdt -q
dataset compact 4 spread
store midastemp.gdt
open AWM.gdt -q
join midastemp.gdt PC6IT_*
remove(midastemp.gdt)
list PCIT = PC6IT_*
setinfo PCIT --midas
</hansl>
Support for wildcards is new in git, and is confined to the case of
importation from a gdt file, for now. Otherwise one would do
join midastemp.gdt PC6IT_m3 PC6IT_m2 PC6IT_m1
In addition, however, join works pretty nicely without the help of a
temporary datafile, provided the high-frequency data start in the
first sub-period (e.g. the first month of a quarter or year): in
that case the "seq" parameter to the "aggr" option does the job:
<hansl>
open AWM.gdt -q
list PCIT = zeros(1,3)
loop i=1..3 -q
vname = sprintf("PC6IT_m%d", i)
join hamilton.gdt @vname --data=PC6IT --aggr=seq:$i
PCIT[4-i] = @vname
endloop
setinfo PCIT --midas
</hansl>
I'll update the gretl-midas doc with one or both of these examples.
I believe that any real solution would also have to cover the
real-time data aspect, because forecasting in real time is going to
be an important application of mixed-frequency data. At first glance
this would suggest join, since there is an entire chapter on it for
real-time data.
That makes sense, yes.
Allin
2:55 p.m.
Am 12.01.2018 um 21:21 schrieb Allin Cottrell:
In addition, however, join works pretty nicely without the help of a
temporary datafile, provided the high-frequency data start in the first
sub-period
<hansl>
open AWM.gdt -q
list PCIT = zeros(1,3)
loop i=1..3 -q
vname = sprintf("PC6IT_m%d", i)
join hamilton.gdt @vname --data=PC6IT --aggr=seq:$i
PCIT[4-i] = @vname
endloop
setinfo PCIT --midas
</hansl>
I'll update the gretl-midas doc with one or both of these examples.
Hm I don't know, to me this is far from user-friendly. The restriction
about the first subperiod is not innocuous, either. Again, compare with
the example in the midas docs for putting together the data from a gretl
database (and also in the GUI, I've tried that right now).
In your earlier email you wrote that the intentional compact-then-match
ordering would be relatively straightforward to implement, although
perhaps not the cleanest way. This still looks like the most attractive
way to me.
Apart from that, I don't really think a temporary file is a problem per
se. In principle I guess a wrapper for the 7 or 8 lines of code that do
the compacting and then joining (or appending) could also be done as a
new hansl function in the extra package.
(BTW, slightly off-topic about the database documentation: The doc for
the 'store' command says about databases: "At present this option is
available only for annual, quarterly or monthly time-series data." The
database spec in the manual also mentions weekly and daily data I
believe. Is on of these infos out of date?)
cheers,
sven
1:30 a.m.
On Sun, 14 Jan 2018, Sven Schreiber wrote:
Am 12.01.2018 um 21:21 schrieb Allin Cottrell:
> In addition, however, join works pretty nicely without the help of a
> temporary datafile, provided the high-frequency data start in the first
> sub-period
> <hansl>
> open AWM.gdt -q
> list PCIT = zeros(1,3)
> loop i=1..3 -q
> vname = sprintf("PC6IT_m%d", i)
> join hamilton.gdt @vname --data=PC6IT --aggr=seq:$i
> PCIT[4-i] = @vname
> endloop
> setinfo PCIT --midas
> </hansl>
>
> I'll update the gretl-midas doc with one or both of these examples.
Hm I don't know, to me this is far from user-friendly. The restriction
about the first subperiod is not innocuous, either.
Granted. The "aggr:seq" option looked quite promising but I agree
it's probably too error-prone. Plus, having to invoke "join" m times
(m being the frequency ratio) is not very nice either.
In your earlier email you wrote that the intentional
compact-then-match ordering would be relatively straightforward to
implement, although perhaps not the cleanest way. This still looks
like the most attractive way to me.
Maybe so, but I'm interested to see if we can inflect the "aggr"
apparatus in "join" such that it handles an n:n join. This may be
too complicated but it might be worth pursuing. The syntax I have in
mind is (provisionally, experimentally) something like
open awm.gdt
join hamilton.gdt PIT* --data=PC6IT
Gretl could internally determine that awm is quarterly and hamilton
monthly, and hence take PC6IT as the source while "PIT*" implictly
expands to PIT_m3, PIT_m2 and PIT_m1 as the target series. (Maybe,
if this can be made to work, it should be conditional on a --spread
or --midas option to "join").
(BTW, slightly off-topic about the database documentation: The doc
for the 'store' command says about databases: "At present this
option is available only for annual, quarterly or monthly
time-series data." The database spec in the manual also mentions
weekly and daily data I believe. Is on of these infos out of
date?)
Hmm, not sure. I think that daily series (at least) are supposed to
be supported in gretl databases. But I can't remember if there's any
extant database that includes daily data; and if there is, I'm sure
it hasn't been tested in a long time. The "store" doc is
deliberately conservative; I don't know offhand if it's too
conservative.
Allin
11:42 a.m.
Am 15.01.2018 um 02:30 schrieb Allin Cottrell:
On Sun, 14 Jan 2018, Sven Schreiber wrote:
> Am 12.01.2018 um 21:21 schrieb Allin Cottrell:
Maybe so, but I'm interested to see if we can inflect the
"aggr"
apparatus in "join" such that it handles an n:n join.
open awm.gdt
join hamilton.gdt PIT* --data=PC6IT
Hmm, perhaps --filter could also be used with the canonical dates? For
example for PIT_m2 we want to have those rows which have an ISO date
with Feb 1st, like this: ????0201
So one could check wether: (rowdate - 201) % 10000 == 0
Of course, a lot of special casing would be necessary with the different
frequencies...
About databases:
Hmm, not sure. I think that daily series (at least) are supposed to
be
supported in gretl databases.
Yes as per appendix A.3 of the guide. (Weekly isn't mentioned there.)
But I can't remember if there's any extant
database that includes daily data;
I couldn't find any.
been tested in a long time. The "store" doc is deliberately
conservative; I don't know offhand if it's too conservative.
I get error messages if I try to store weekly or daily data to a
database. So the 'store' doc looks right. The question then is how to
actually create a database with daily data?
thanks,
sven
4:33 p.m.
On Mon, 15 Jan 2018, Sven Schreiber wrote:
About databases:
> Hmm, not sure. I think that daily series (at least) are supposed to be
> supported in gretl databases. [Allin]
Yes as per appendix A.3 of the guide. (Weekly isn't mentioned there.)
> But I can't remember if there's any extant database that includes
> daily data; [Allin]
I couldn't find any.
I get error messages if I try to store weekly or daily data to a database. So
the 'store' doc looks right. The question then is how to actually create a
database with daily data?
It seems that daily series in gretl databases was an idea that was
only partially implemented and then never really developed. In today's
git and snapshots there are some relevant updates. You can now "store"
daily data in gretl db format, and read them back into a blank
workspace, or a matching daily workspace, or (via --compact=spread)
into a monthly workspace. At this point the GUI is only partially
synced with these changes.
Here's "proof of concept":
<hansl>
open djclose.gdt
# create daily database
store djc.bin djclose
nulldata 50
# establish monthly dataset
setobs 12 1980:01
open djc.bin
# import daily data
data djclose --compact=spread
list DJC = djclose_*
setinfo DJC --midas
print DJC --midas
</hansl>
Allin
6 p.m.
Am 16.01.2018 um 17:33 schrieb Allin Cottrell:
It seems that daily series in gretl databases was an idea that was
only
partially implemented and then never really developed. In today's git
and snapshots there are some relevant updates. You can now "store" daily
data in gretl db format, and read them back into a blank workspace, or a
matching daily workspace, or (via --compact=spread) into a monthly
workspace. At this point the GUI is only partially synced with these
changes.
Very good. Some things I noticed with this latest snapshot:
- in the idx file of the database the date format is ISO style:
YYYY-MM-DD, whereas appendix A.3 says one must use YYYY/MM/DD. I guess
the doc needs updating, or are there existing databases with that format?
- I can store 5-day and 7-day daily data alright.
- I can also store 6-day daily data (an idea which I got from the setobs
documentation) which then gets an "S" code, presumably for "special".
I
can even reimport that series into a fresh gretl workspace, and to my
surprise gretl recognizes it as 6-day data. I'm actually not sure where
it got the information from -- a clever combination of start and end
dates and the num of obs? Or is "S" actually for Six?
- The doc says the database description in idx can only be 64 characters
long. Why? Even the description of each series can be 128 chars.
Thanks,
sven
6:08 p.m.
On Tue, 16 Jan 2018, Sven Schreiber wrote:
Am 16.01.2018 um 17:33 schrieb Allin Cottrell:
> It seems that daily series in gretl databases was an idea that was only
> partially implemented and then never really developed. In today's git and
> snapshots there are some relevant updates. You can now "store" daily data
> in gretl db format, and read them back into a blank workspace, or a
> matching daily workspace, or (via --compact=spread) into a monthly
> workspace. At this point the GUI is only partially synced with these
> changes.
Very good. Some things I noticed with this latest snapshot:
- in the idx file of the database the date format is ISO style: YYYY-MM-DD,
whereas appendix A.3 says one must use YYYY/MM/DD. I guess the doc needs
updating, or are there existing databases with that format?
The doc is now updated in favour of YYYY-MM-DD. But the old YYYY/MM/DD
is still accepted just in case anyone has created such a file.
- I can store 5-day and 7-day daily data alright.
- I can also store 6-day daily data (an idea which I got from the setobs
documentation) which then gets an "S" code, presumably for "special".
I can
even reimport that series into a fresh gretl workspace, and to my surprise
gretl recognizes it as 6-day data. I'm actually not sure where it got the
information from -- a clever combination of start and end dates and the num
of obs? Or is "S" actually for Six?
Yes, "S" stands for 6. That's now in the doc too.
- The doc says the database description in idx can only be 64
characters long. Why? Even the description of each series can be 128
chars.
Not sure, I can take a look at that.
BTW, there was some collateral breakage in the first snapshots I put
out today. The very latest ones should be better.
Allin
2531
days inactive
2536
days old
11 comments
3 participants
participants (3)
-
Allin Cottrell -
Riccardo (Jack) Lucchetti -
Sven Schreiber