9:52 a.m.
Hi,
attached is a monthly small dataset (with missings, if that matters).
Trying to reduce the data frequency ("compacting") to quarterly in a
Midas way ("separate series") gretl crashes every time. On Windows with
snapshot Oct 3rd.
Apart from that, a minor suggestion: In the data compacting window, the
label "spread to separate series" (or similar, ad-hoc re-translated from
German) might get an extra hint: "(Midas)".
thanks,
sven
11:04 a.m.
Am 05.10.2017 um 11:52 schrieb Sven Schreiber:
attached is a monthly small dataset (with missings, if that matters).
Trying to reduce the data frequency ("compacting") to quarterly in a
Midas way ("separate series") gretl crashes every time. On Windows with
snapshot Oct 3rd.
Same thing when I try to go via a database (.bin).
Or with gretl-2017c.
thanks,
sven
3:22 p.m.
On Thu, 5 Oct 2017, Sven Schreiber wrote:
Hi,
attached is a monthly small dataset (with missings, if that matters). Trying
to reduce the data frequency ("compacting") to quarterly in a Midas way
("separate series") gretl crashes every time. On Windows with snapshot Oct
3rd.
OK, I see what's wrong (not to do with missing values, to do with the
fact that the monthly dataset started at an offset into the first
usable quarter). That's now fixed in git and snapshots.
Apart from that, a minor suggestion: In the data compacting window,
the label
"spread to separate series" (or similar, ad-hoc re-translated from German)
might get an extra hint: "(Midas)".
That's probably worth doing, though it's not done yet.
Allin
6:11 p.m.
Am 05.10.2017 um 17:22 schrieb Allin Cottrell:
OK, I see what's wrong (not to do with missing values, to do with
the
fact that the monthly dataset started at an offset into the first usable
quarter). That's now fixed in git and snapshots.
Great, I can confirm it works now.
thanks,
sven
7:51 a.m.
Am 05.10.2017 um 20:11 schrieb Sven Schreiber:
Am 05.10.2017 um 17:22 schrieb Allin Cottrell:
> OK, I see what's wrong (not to do with missing values, to do with the
> fact that the monthly dataset started at an offset into the first
> usable quarter). That's now fixed in git and snapshots.
Great, I can confirm it works now.
Another little Midas thing, GUI-related: Take the attached file which is
now successfully compacted to quarterly, but still has the monthly Midas
series in it.
Then do:
series check = spread12_3_m3(-1)
The series is correctly created with ID 10, for example "eval
varname(10)" works OK. Next to the original series with ID 7 you get the
little + button that indicates presence of lagged series. However, you
cannot do anything with the button, apparently because spread12_3_m3 is
part of a Midas collection/list.
This is of course very minor and probably low priority. Should I put it
on the tracker?
thanks,
sven
4:46 p.m.
New subject: crash when trying Midas forecasting (was: -compacting)
Am 05.10.2017 um 20:11 schrieb Sven Schreiber:
Am 05.10.2017 um 17:22 schrieb Allin Cottrell:
> OK, I see what's wrong (not to do with missing values, to do with the
> fact that the monthly dataset started at an offset into the first
> usable quarter). That's now fixed in git and snapshots.
Great, I can confirm it works now.
Hi, I've encountered another crash with the latest (yesterday's)
snapshot. I suspect it has to do with using several Midas terms for
forecasting -- I vaguely seem to remember now that it isn't supported
yet, and sometimes I just get an error message. (Although the message
"missing values encountered" seems misleading.)
But in the following case there's a hard crash, apparently triggered by
using different types of specifications (nealmon/umidas) in one equation:
# debug: for direct comparison
midasreg dep const ownlag otherQfreq; \
mds(dIP, start, hilagU, "nealmon") \
mds(dO, start, hilagU, "umidas") \
mds(Spr, startSpr, stopSpr, "umidas")
fcast ($t2+1) endFC
# end debug
Thanks,
sven
5:50 p.m.
New subject: crash when trying Midas forecasting (was: -compacting)
On Fri, 6 Oct 2017, Sven Schreiber wrote:
Am 05.10.2017 um 20:11 schrieb Sven Schreiber:
> Am 05.10.2017 um 17:22 schrieb Allin Cottrell:
>
>> OK, I see what's wrong (not to do with missing values, to do with the fact
>> that the monthly dataset started at an offset into the first usable
>> quarter). That's now fixed in git and snapshots.
>
> Great, I can confirm it works now.
Hi, I've encountered another crash with the latest (yesterday's) snapshot. I
suspect it has to do with using several Midas terms for forecasting -- I
vaguely seem to remember now that it isn't supported yet, and sometimes I
just get an error message. (Although the message "missing values encountered"
seems misleading.)
It's supposed to be supported, though it hasn't been tested much and I
wouldn't really expect it to work well (excessive collinearity?).
But in the following case there's a hard crash, apparently
triggered
by using different types of specifications (nealmon/umidas) in one
equation:
# debug: for direct comparison
midasreg dep const ownlag otherQfreq; \
mds(dIP, start, hilagU, "nealmon") \
mds(dO, start, hilagU, "umidas") \
mds(Spr, startSpr, stopSpr, "umidas")
fcast ($t2+1) endFC
# end debug
Mixing specifications, again, ought to work in principle. And of
course a crash is always a bug.
Could you maybe send me your data offlist?
Allin
7:40 p.m.
New subject: crash when trying Midas forecasting (was: -compacting)
On Fri, Oct 6, 2017 at 1:50 PM, Allin Cottrell <cottrell(a)wfu.edu> wrote:
> [I]n the following case there's a hard crash, apparently
triggered by
> using different types of specifications (nealmon/umidas) in one equation:
>
> # debug: for direct comparison
> midasreg dep const ownlag otherQfreq; \
> mds(dIP, start, hilagU, "nealmon") \
> mds(dO, start, hilagU, "umidas") \
> mds(Spr, startSpr, stopSpr, "umidas")
> fcast ($t2+1) endFC
> # end debug
Mixing specifications, again, ought to work in principle. And of course a
crash is always a bug.
Could you maybe send me your data offlist?
Never mind sending the data, I can reproduce a crash with an example
of my own. It's specific to combining an mds() "umidas" term with other
midas terms (mdsl() "umidas" terms can be combined OK). I can see
what's going wrong, and will fix it once I've sent out the homework
exercise that I promised my macroeconomics class!
Allin
8:18 p.m.
New subject: crash when trying Midas forecasting (was: -compacting)
On Fri, Oct 6, 2017 at 3:40 PM, Cottrell, Allin <cottrell(a)wfu.edu> wrote:
On Fri, Oct 6, 2017 at 1:50 PM, Allin Cottrell
<cottrell(a)wfu.edu> wrote:
>> [I]n the following case there's a hard crash, apparently triggered by
>> using different types of specifications (nealmon/umidas) in one equation:
>>
>> # debug: for direct comparison
>> midasreg dep const ownlag otherQfreq; \
>> mds(dIP, start, hilagU, "nealmon") \
>> mds(dO, start, hilagU, "umidas") \
>> mds(Spr, startSpr, stopSpr, "umidas")
>> fcast ($t2+1) endFC
>> # end debug
>
> Mixing specifications, again, ought to work in principle. And of course a
> crash is always a bug.
>
> Could you maybe send me your data offlist?
Never mind sending the data, I can reproduce a crash with an example
of my own.
That's now fixed in git, revised snapshots will follow.
Thanks to Sven for beating out bugs in midasreg. I'll follow up by
running some more tests of my own.
Allin
9:21 p.m.
New subject: crash when trying Midas forecasting
Am 06.10.2017 um 19:50 schrieb Allin Cottrell:
On Fri, 6 Oct 2017, Sven Schreiber wrote:
> snapshot. I suspect it has to do with using several Midas terms
for
> forecasting -- I vaguely seem to remember now that it isn't supported
> yet, and sometimes I just get an error message. (Although the message
> "missing values encountered" seems misleading.)
It's supposed to be supported, though it hasn't been tested much and I
wouldn't really expect it to work well (excessive collinearity?).
Well the pure UMidas variant works, so I don't see why the problem
should be so severe as to result in failure with the Almon spec.
Actually here's a reproducible example for the other error I'm getting
("found missing values"):
<hansl>
open gdp_midas.gdt
list H1 = ld_payems_m?
list H2 = ld_indpro_m?
list L1 = hflags(1, 5, H1)
list L2 = hflags(1, 5, H2)
smpl +0 -3
midasreg ld_qgdp const ld_qgdp(-1) ; \
mdsl(L1, "nealmon") mdsl(L2, "nealmon")
fcast ($t2+1) ($t2+2) --static
</hansl>
BTW, when I write "fcast ($t2 + 1) ($t2 + 2)" with some blanks instead,
I get a different (unrelated, I think) error.
thanks,
sven
12:12 a.m.
New subject: crash when trying Midas forecasting
On Fri, 6 Oct 2017, Sven Schreiber wrote:
Am 06.10.2017 um 19:50 schrieb Allin Cottrell:
> On Fri, 6 Oct 2017, Sven Schreiber wrote:
>> snapshot. I suspect it has to do with using several Midas terms for
>> forecasting -- I vaguely seem to remember now that it isn't supported yet,
>> and sometimes I just get an error message. (Although the message "missing
>> values encountered" seems misleading.)
>
> It's supposed to be supported, though it hasn't been tested much and I
> wouldn't really expect it to work well (excessive collinearity?).
Well the pure UMidas variant works, so I don't see why the problem should be
so severe as to result in failure with the Almon spec.
Actually here's a reproducible example for the other error I'm getting
("found missing values"):
<hansl>
open gdp_midas.gdt
list H1 = ld_payems_m?
list H2 = ld_indpro_m?
list L1 = hflags(1, 5, H1)
list L2 = hflags(1, 5, H2)
smpl +0 -3
midasreg ld_qgdp const ld_qgdp(-1) ; \
mdsl(L1, "nealmon") mdsl(L2, "nealmon")
fcast ($t2+1) ($t2+2) --static
</hansl>
The crash you reported in a prior posting was just a case of simple
oversight (with regard to the printing of midasreg results), easily
fixed by (pretty much) a one-liner in the underlying code.
This, however, is more substantive: at present we're not saving
enough information on a midasreg model to generate a forecast, when
the model comprises more than one mds() or mdsl() term. I need to
think a bit about how best to arrange that.
BTW, when I write "fcast ($t2 + 1) ($t2 + 2)" with some
blanks
instead, I get a different (unrelated, I think) error.
OK, we'll get to that later. But I think it may have been a mistake
to allow arbitrary expressions for the start and end of the forecast
range in the "fcast" command. We'd have a better chance of avoiding
errors if we insisted that these limits be (a) integer observation
numbers in numerical form, (b) the names of existing scalar
variables containing such values, or (c) observation strings such as
"2010:4".
But then, maybe if we fix the primary problem the secondary one will
go away.
Allin
1:57 p.m.
New subject: crash when trying Midas forecasting
Am 07.10.2017 um 02:12 schrieb Allin Cottrell:
On Fri, 6 Oct 2017, Sven Schreiber wrote:
This, however, is more substantive: at present we're not saving
enough
information on a midasreg model to generate a forecast, when the model
comprises more than one mds() or mdsl() term. I need to think a bit
about how best to arrange that.
Really? When I look at $model.midas_info and its member bundles
(here: midas_info[1] and midas_info[2], in general up to the number of
Midas terms) it seems there's everything one needs, in combination with
$model.coeff.
It would probably be a bit tedious but apparently doable even from the
hansl side, with the help of mlincomb() -- just talking about point
forecasts here now.
Do you want me to attempt a prototype?
> BTW, when I write "fcast ($t2 + 1) ($t2 + 2)" with some
blanks
> instead, I get a different (unrelated, I think) error.
OK, we'll get to that later. But I think it may have been a mistake to
allow arbitrary expressions for the start and end of the forecast range
in the "fcast" command. We'd have a better chance of avoiding errors if
we insisted that these limits be (a) integer observation numbers in
numerical form, (b) the names of existing scalar variables containing
such values, or (c) observation strings such as "2010:4".
Well, maybe, in principle I wouldn't mind that restriction too much.
However, with the observation strings I recall having some problems in
the past. And there's the ambiguity about annual data year versus obs
numbers. So what really seems important to support is the use of
obsnum() expressions right in the fcast command, and perhaps also
obslabel(). Otherwise it would feel a bit lame, to be honest.
thanks,
sven
3:17 p.m.
New subject: crash when trying Midas forecasting
On Sat, 7 Oct 2017, Sven Schreiber wrote:
Am 07.10.2017 um 02:12 schrieb Allin Cottrell:
> On Fri, 6 Oct 2017, Sven Schreiber wrote:
> This, however, is more substantive: at present we're not saving enough
> information on a midasreg model to generate a forecast, when the model
> comprises more than one mds() or mdsl() term. I need to think a bit about
> how best to arrange that.
Really? When I look at $model.midas_info and its member bundles
(here: midas_info[1] and midas_info[2], in general up to the number of Midas
terms) it seems there's everything one needs, in combination with
$model.coeff.
It would probably be a bit tedious but apparently doable even from the hansl
side, with the help of mlincomb() -- just talking about point forecasts here
now.
Do you want me to attempt a prototype?
Thanks for the offer, but this is now fixed in git and snapshots.
The thing is that for forecasting we need the "gross" MIDAS
coefficients (hfslope, if any, times the weights implied by the
hyperparams, for each midas term). Yes, that could be figured out
using the data in the "midas_info" bundles, but I think it's easier
to compute it when the model is estimated and stick the consolidated
vector onto the model. We now do that (under the name "hfb").
>> BTW, when I write "fcast ($t2 + 1) ($t2 + 2)" with
some blanks instead, I
>> get a different (unrelated, I think) error.
>
> OK, we'll get to that later. But I think it may have been a mistake to
> allow arbitrary expressions for the start and end of the forecast range in
> the "fcast" command. We'd have a better chance of avoiding errors if we
> insisted that these limits be (a) integer observation numbers in numerical
> form, (b) the names of existing scalar variables containing such values, or
> (c) observation strings such as "2010:4".
Well, maybe, in principle I wouldn't mind that restriction too much. However,
with the observation strings I recall having some problems in the past. And
there's the ambiguity about annual data year versus obs numbers. So what
really seems important to support is the use of obsnum() expressions right in
the fcast command, and perhaps also obslabel(). Otherwise it would feel a bit
lame, to be honest.
Well, I'm not sure if it fully meets your request, but we now parse
the "startobs" and "endobs" arguments to fcast more liberally (in
git, not yet snapshots). So your variant with parenthesized
expressions containing spaces will now work.
Allin
10:07 a.m.
New subject: crash when trying Midas forecasting
Am 07.10.2017 um 17:17 schrieb Allin Cottrell:
On Sat, 7 Oct 2017, Sven Schreiber wrote:
> Am 07.10.2017 um 02:12 schrieb Allin Cottrell:
>> On Fri, 6 Oct 2017, Sven Schreiber wrote:
>
>> This, however, is more substantive: at present we're not saving
>> enough information on a midasreg model to generate a forecast, when
>> the model comprises more than one mds() or mdsl() term. I need to
>> think a bit about how best to arrange that.
Thanks for the offer, but this is now fixed in git and snapshots. The
thing is that for forecasting we need the "gross" MIDAS coefficients
(hfslope, if any, times the weights implied by the hyperparams, for each
midas term).
I'm not sure if this is now correct, I seem to be getting some strange
results for the Umidas case, where I _believe_ they were different (more
plausible) before. I'll have to check, and will report back.
"midas_info" bundles, but I think it's easier to
compute it when the
model is estimated and stick the consolidated vector onto the model. We
now do that (under the name "hfb").
Will compare with that one, too.
thanks,
sven
11:29 a.m.
New subject: crash when trying Midas forecasting
Am 09.10.2017 um 12:07 schrieb Sven Schreiber:
Am 07.10.2017 um 17:17 schrieb Allin Cottrell:
> Thanks for the offer, but this is now fixed in git and snapshots.
The
> thing is that for forecasting we need the "gross" MIDAS coefficients
> (hfslope, if any, times the weights implied by the hyperparams, for
> each midas term).
I'm not sure if this is now correct, I seem to be getting some strange
results for the Umidas case, where I _believe_ they were different (more
plausible) before. I'll have to check, and will report back.
Well, for example I am getting this bogus type of estimation output with
yesterday's snapshot:
<output>
=== normalized exponential Almon ===
Konvergenz erreicht nach 67 Iterationen
Modell 4: MIDAS (NKQ), benutze die Beobachtungen 1991:4-2009:4 (T = 73)
L-BFGS-B mit bedingten KQ benutzt
Abhängige Variable: dep
Schätzung Std.-fehler t-Quotient p-Wert
-------------------------------------------------------------
const 0.00310599 0.000813343 3.819 0.0003 ***
dep_1 −0.164985 0.118316 −1.394 0.1682
const 0.277335 0.0991822 2.796 0.0069 ***
...
</output>
Note the double "const" entry.
With "almonp", "betan" and "beta0" similar (sometimes
slightly
different) problems.
As I said, I suspect this is more than just a labeling problem, given
the resulting forecasts. (not shown here)
thanks,
sven
6:05 p.m.
New subject: crash when trying Midas forecasting
On Mon, 9 Oct 2017, Sven Schreiber wrote:
Am 09.10.2017 um 12:07 schrieb Sven Schreiber:
> Am 07.10.2017 um 17:17 schrieb Allin Cottrell:
>> Thanks for the offer, but this is now fixed in git and snapshots. The
>> thing is that for forecasting we need the "gross" MIDAS coefficients
>> (hfslope, if any, times the weights implied by the hyperparams, for
>> each midas term).
>
> I'm not sure if this is now correct, I seem to be getting some strange
> results for the Umidas case, where I _believe_ they were different (more
> plausible) before. I'll have to check, and will report back.
Well, for example I am getting this bogus type of estimation output with
yesterday's snapshot:
<output>
=== normalized exponential Almon ===
Konvergenz erreicht nach 67 Iterationen
Modell 4: MIDAS (NKQ), benutze die Beobachtungen 1991:4-2009:4 (T = 73)
L-BFGS-B mit bedingten KQ benutzt
Abhängige Variable: dep
Schätzung Std.-fehler t-Quotient p-Wert
-------------------------------------------------------------
const 0.00310599 0.000813343 3.819 0.0003 ***
dep_1 −0.164985 0.118316 −1.394 0.1682
const 0.277335 0.0991822 2.796 0.0069 ***
...
</output>
Can you tell me what the midasreg specification looks like? I'm not
seeing anything like that in the examples I'm running, but I guess
they're not general enough.
Allin
6:17 p.m.
New subject: crash when trying Midas forecasting
Am 09.10.2017 um 20:05 schrieb Allin Cottrell:
On Mon, 9 Oct 2017, Sven Schreiber wrote:
> <output>
> === normalized exponential Almon ===
> Konvergenz erreicht nach 67 Iterationen
>
> Modell 4: MIDAS (NKQ), benutze die Beobachtungen 1991:4-2009:4 (T = 73)
> L-BFGS-B mit bedingten KQ benutzt
> Abhängige Variable: dep
>
> Schätzung Std.-fehler t-Quotient p-Wert
> -------------------------------------------------------------
> const 0.00310599 0.000813343 3.819 0.0003 ***
> dep_1 −0.164985 0.118316 −1.394 0.1682
> const 0.277335 0.0991822 2.796 0.0069 ***
> ...
> </output>
Can you tell me what the midasreg specification looks like? I'm not
seeing anything like that in the examples I'm running, but I guess
they're not general enough.
Right, sorry. Here's a relevant script portion:
<hansl>
# debug
list otherQfreq print
list dIPL print
list dOL print
list SprL print
print "=== normalized exponential Almon ==="
midasreg dep const ownlag otherQfreq; \
mdsl(dIPL, "nealmon") \
mdsl(dOL, "nealmon") \
mdsl(SprL, "nealmon") # mdsl(WL, "nealmon")
</hansl>
The "ownlag" is a list that has just dep_1 in it, this seems to work
alright.
And for completeness the output of the debug list printing is as
follows, with "otherQfreq" on top also just holding a single member:
giphat_m2_0
gip_m1_0 gip_m3_1 gip_m2_1 gip_m1_1 gip_m3_2 gip_m2_2 gip_m1_2 \
gip_m3_3 gip_m2_3
ld_orders_m1_0 ld_orders_m3_1 ld_orders_m2_1 ld_orders_m1_1 ld_orders_m3_2 \
ld_orders_m2_2 ld_orders_m1_2 ld_orders_m3_3 ld_orders_m2_3
spread12_3_m2_0 spread12_3_m1_0 spread12_3_m3_1 spread12_3_m2_1 \
spread12_3_m1_1 spread12_3_m3_2 spread12_3_m2_2 spread12_3_m1_2 \
spread12_3_m3_3 spread12_3_m2_3
thanks,
sven
9:15 p.m.
New subject: crash when trying Midas forecasting
On Mon, 9 Oct 2017, Sven Schreiber wrote:
Am 09.10.2017 um 20:05 schrieb Allin Cottrell:
> On Mon, 9 Oct 2017, Sven Schreiber wrote:
>> <output>
>> === normalized exponential Almon ===
>> Konvergenz erreicht nach 67 Iterationen
>>
>> Modell 4: MIDAS (NKQ), benutze die Beobachtungen 1991:4-2009:4 (T = 73)
>> L-BFGS-B mit bedingten KQ benutzt
>> Abhängige Variable: dep
>>
>> Schätzung Std.-fehler t-Quotient p-Wert
>> -------------------------------------------------------------
>> const 0.00310599 0.000813343 3.819 0.0003 ***
>> dep_1 −0.164985 0.118316 −1.394 0.1682
>> const 0.277335 0.0991822 2.796 0.0069 ***
>> ...
>> </output>
>
> Can you tell me what the midasreg specification looks like? I'm not
> seeing anything like that in the examples I'm running, but I guess
> they're not general enough.
Right, sorry. Here's a relevant script portion:
<hansl>
# debug
list otherQfreq print
list dIPL print
list dOL print
list SprL print
print "=== normalized exponential Almon ==="
midasreg dep const ownlag otherQfreq; \
mdsl(dIPL, "nealmon") \
mdsl(dOL, "nealmon") \
mdsl(SprL, "nealmon") # mdsl(WL, "nealmon")
</hansl>
Thanks, Sven. The problem (or at least, the most obvious problem)
was that low-frequency regressors that appear after the lagged
dependent variable (if present) in the midasreg specification were
not being treated correctly. That should now be fixed in git.
As for more general testing of the case where multiple MIDAS terms
appear in the specification, that's a little tricky. Ghysels' Matlab
code accepts only one MIDAS term, and while midasr accepts multiple
terms there's the problem that they define high-frequency lags
differently (from Matlab and gretl). Nonetheless, it should be
possible to set up a comparison against midasr, with a little
effort. I'll give that a try.
Allin
3:54 p.m.
New subject: crash when trying Midas forecasting
Am 09.10.2017 um 23:15 schrieb Allin Cottrell:
Thanks, Sven. The problem (or at least, the most obvious problem) was
that low-frequency regressors that appear after the lagged dependent
variable (if present) in the midasreg specification were not being
treated correctly. That should now be fixed in git.
Thanks, with the latest snapshot I'm now getting sensible results
(meaning estimates as well as forecasts), both with UMidas as well as
with the parsimonious specifications.
As for more general testing of the case where multiple MIDAS terms
appear in the specification, that's a little tricky. Ghysels' Matlab
code accepts only one MIDAS term, and while midasr accepts multiple
terms there's the problem that they define high-frequency lags
differently (from Matlab and gretl). Nonetheless, it should be possible
to set up a comparison against midasr, with a little effort. I'll give
that a try.
Of course "sensible results" are no proof that they are correct, but I'm
relatively confident. Apart from that, I'm not sure that your scarce
time resources should be used on testing those things.
Any R aficionados out there for whom it would be easy to handle R's
"midasr" package?
[Provided anybody is still following this thread at all...]
thanks,
sven
4:02 p.m.
New subject: crash when trying Midas forecasting
On Tue, 10 Oct 2017, Sven Schreiber wrote:
Any R aficionados out there for whom it would be easy to handle
R's "midasr"
package?
[Provided anybody is still following this thread at all...]
Can you see those little dots on the horizon? :D
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
2692
days inactive
2697
days old
19 comments
4 participants
participants (4)
-
Allin Cottrell -
Cottrell, Allin
-
Riccardo (Jack) Lucchetti -
Sven Schreiber