7:07 p.m.
Sven has raised the question of the handling of daily data in gretl;
see the threads starting from
http://lists.wfu.edu/pipermail/gretl-users/2014-May/010037.html
I'm glad of that: it's time we clarified what we do now, and what we
should do in future. (But please note, I'm mostly talking here about
5-day financial-market data; other sorts of daily data might require
different handling.)
Sorry, this is long, but I'd encourage those who work with daily
data to read on...
First a minor point in relation to Sven's example: I think the
Bundesbank is in fact unusual in including blank weekends in
business-day data files. At least, that's not the practice of the
Federal Reserve, the Bank of England, the Banque de France, the
Banca d'Italia, the Sveriges Riksbank... (at which point I got tired
of googling).
Anyway, it's (now) easy enough to strip out weekends, which leaves
the more interesting question of how to deal with holidays.
I think it's fair to say:
(a) most econometricians who wish to apply time-series methods to
daily financial market data will, most of the time, want to ignore
holidays as well as weekends, treating the data as if these days did
not exist and the actual trading days formed a continuous series,
but
(b) for some purposes it may be important to be able to recover
information on (e.g.) which days were Mondays or which days followed
holidays.
How are these needs best supported by econometric software? I can
see two possibilities:
(1) The storage for 5-day data includes rows for all Mondays to
Fridays (or even all days as per the Bundesbank) -- hence satisfying
point (b) automatically -- and the software provides a mechanism for
skipping non-trading days on demand when estimating models.
(2) The data storage includes only actual trading days -- hence
satisfying point (a) automatically -- but with a record of their
calendar dates, and the software provides means of retrieving the
information under point (b) on demand.
Currently gretl includes a half-hearted gesture towards approach (1)
but de facto mostly relies on approach (2). Let me explain.
When we first introduced support for daily data I initially assumed
that we'd want to store 5-day data including rows for all relevant
days, with NAs for holidays. So in view of point (a) above I put in
place a mechanism for skipping NAs in daily data when doing OLS. But
this never got properly documented, and it was never extended to
other estimators.
What happened? Well, as we started adding examples of daily data to
the gretl package it became apparent that approach (2) is quite
common in practice. See for example the "djclose" dataset from Stock
and Watson and the Bollerlev-Ghysels exchange-rate returns series
(b-g.gdt). Both of these have non-trading days squeezed out of them;
let's call this "compressed" daily data.
The Bollerlev-Ghysels dataset is not the best example, as the
authors did not record the actual dates of the included
observations, only the starting and ending dates. But djclose will
serve as a test case: although it excludes non-trading days the date
of each observation is recorded in its "marker" string and it's
straightforward to retrieve all the information one might want via
gretl's calendrical functions, as illustrated below.
<hansl>
/* analysis of compressed 5-day data */
open djclose.gdt
# get day of week and "epoch day" number
series wd = weekday($obsmajor, $obsminor, $obsmicro)
series ed = epochday($obsmajor, $obsminor, $obsmicro)
# maybe we want a dummy for Mondays?
series monday = wd == 1
# find the "delta days" between observations
series delta = diff(ed)
# the "standard" delta days in absence of holidays:
# three for Mondays, otherwise one
series std_delta = wd == 1 ? 3 : 1
# create a dummy for days following holidays
series posthol = delta > std_delta
# take a look...
print wd monday delta posthol --byobs
</hansl>
Here's a proposal for regularizing our handling of daily data. In
brief, it's this: scrap our gesture towards what I called approach
(1) above, and beef up our support for approach (2).
Why get rid of the mechanism for automatically skipping NAs in daily
data for OLS? Because it's anomalous that it only works for OLS, it
would be a lot of work to provide this mechanism for all estimators,
and anyway it probably should not be automatic: ignoring NAs when
they're present in the dataset should require some user
intervention.
By beefing up approach (2) I mean providing easy means of converting
between "uncompressed" and "compressed" daily data. We already
support both variants, but (a) given an uncompressed daily sequence
it should be easy for the user to squeeze out NAs if she thinks
that's appropriate for estimation purposes, and (b) it might be
useful in some contexts to be able to reconstitute the full calendar
sequence from a compressed dataset such as djclose.
Such conversion is possible via "low-level" hansl, but not
convenient. I've therefore added the following two things in
CVS/snapshots:
(1) If you apply an "smpl" restriction to a daily dataset, we try to
reconstitute a useable daily time-series. If it has gaps, we record
the specific dates of the included observations. At present this is
subject to two conditions, which are open to discussion.
(i) Define the "delta" of a given daily observation as the epoch day
(= 1 for the first of January in 1 AD) of that observation minus the
epoch day of the previous one. So, for example, in the case of
complete 7-day data the delta will always be 1. With complete 5-day
data the delta will be 3 for Mondays and 1 for Tuesdays through
Fridays. The first condition on converting from "full" data to
something like djclose.gdt (dated daily data with gaps) is that the
maximum daily delta is less than 10.
(ii) The "smpl" restriction in question may involve throwing away
"empty" weekends; this will lose about 2/7 of the observations and
preserve about 5/7. Allowing for this, we then require that the
number of observations in the sub-sample is at least 90 percent of
the maximum possible. Or in other words we're allowing up to 10
percent loss of observations due to holidays. That's generous --
perhaps too generous?
(The point of these restrictions is to avoid "pretending" that a
seriously gappy daily sequence -- much gappier than could be
accounted for by trading holidays -- can be treated as if it were a
continuous time series for econometric purposes.)
(2) Second thing added: a new trope for the "dataset" command,
namely
dataset pad-daily <days-in-week>
This will pad out a dataset such as djclose, adding in NAs for
holidays and (if the days-in-week parameter is 7) for weekends too.
I'm not sure if this second thing is worth keeping and documenting,
but for now it permits a test of the whole apparatus by
round-tripping. Here's an example, supposing we're starting from
data on a complete 7-day calendar, but with empty weekends and
all-NA rows for holidays (as in Sven's Bundesbank data):
<hansl>
open <seven-day-data>
outfile orig.txt --write
print --byobs
outfile --close
smpl --no-missing --permanent
outfile compressed.txt --write
print --byobs
outfile --close
dataset pad-daily 7
outfile reconstructed.txt --write
print --byobs
outfile --close
string diffstr = $(diff orig.txt reconstructed.txt)
printf "diffstr = '%s'\n", diffstr
</hansl>
So if the round trip is successful, diffstr should be empty. Ah, but
with Sven's data it's not quite empty. What's the problem? It's with
the logic of --no-missing, which excludes all rows on which there's
at least one NA. What we really want, to skip holidays, is to
exclude all and only those rows on which all of our daily variables
are NA. That's feasible via raw hansl, but not so convenient. So one
more modification to "smpl" in CVS: add an option --no-all-missing
(the name may be debatable). Substitute --no-all-missing for
--no-missing in the script above and the difference between orig.txt
and reconstructed.txt really is null.
If you don't have a handy Bundesbank-style data file (though it's
not hard to fake one), here's another round-trip test, in the other
direction: we pad out djclose then shrink it again.
<hansl>
open djclose.gdt -q
outfile orig.txt --write
print --byobs
outfile --close
dataset pad-daily 5
outfile padded.txt --write
print --byobs
outfile --close
smpl --no-all-missing --permanent
outfile reconstructed.txt --write
print --byobs
outfile --close
string diffstr = $(diff orig.txt reconstructed.txt)
printf "diffstr = '%s'\n", diffstr
</hansl>
The use of the --permanent option in the round-trip scripts is just
to ensure that all vestiges of the original data are destroyed
before the reconstruction takes place. In "normal usage" one could
just do
<hansl fragment="true">
open <seven-day-data>
smpl --no-all-missing
</hansl>
then carry out econometric analysis without tripping over NAs.
Allin