8:35 p.m.
Greetings to all,
Being a very inexperienced Gretl user, I ran into an issue which must have a simple
solution, however I could find it nowhere.
My question is the following :
In a time series which comprises daily entries over 10 years, how may I select a certain
period of each year, i.e. only March-April for instance ?
Up to now I have had to resort to a most inefficient solution, although it seems to work,
whereby a given variable time (containing only the dates of the entries under a format
20040101, which Gretl doesn't recognise as a date) was used to open a new variable
month=int((time-10000*int(time/10000))/100)
This works, in that I now have a variable returning the number of each month. I then
proceeded to restrict based on the criterion month >= 2 && month <= 4
However the result of this is that I have a series which is not chronological any more. If
I convert it back to a time series, it logically gives me a roughly two-year-long period
with daily entries. Which means I should proceed to isolate the trimester of each year
manually. This is very inefficient and time-consuming.
As this is surely not an over-the-top functionality, I assume there must be a simpler and
more efficient way of selecting a given period within each year, repetitively over several
years. Can someone kindly help me out ?
Hoping my explanations were clear, and thanking all in advance,
Best,
Marc O'Callaghan
11:33 p.m.
On Thu, 10 Dec 2015, Marc O'Callaghan wrote:
Greetings to all,
Being a very inexperienced Gretl user, I ran into an issue which
must have a simple solution, however I could find it nowhere.
My question is the following :
In a time series which comprises daily entries over 10 years, how
may I select a certain period of each year, i.e. only March-April
for instance ? Up to now I have had to resort to a most
inefficient solution, although it seems to work, whereby a given
variable time (containing only the dates of the entries under a
format 20040101, which Gretl doesn't recognise as a date) [...]
Just pointing out, if these dates contained punctuation --
2004-01-01 or 2004/01/01 -- and they're in the first column of
imported data, they'd be recognized OK. In addition, if the
daily data is complete (with regular 5, 6, or 7 days per week) you
can get the dates set in gretl using the "setobs" command, or the
"Dataset structure" item under the "Data" menu. Then the $obsminor
accessor would get you a series of months.
was used to open a new variable
month=int((time-10000*int(time/10000))/100) This works, in that I
now have a variable returning the number of each month. I then
proceeded to restrict based on the criterion month >= 2 && month
<= 4 However the result of this is that I have a series which is
not chronological any more.
The data will still be in chronological order, surely? But they
won't be a time series in the usual sense, since the sequence
contains huge gaps.
If I convert it back to a time series, it logically gives me a
roughly two-year-long period with daily entries. Which means I
should proceed to isolate the trimester of each year manually.
This is very inefficient and time-consuming.
Sorry, I don't understand what exactly it is that's inefficient and
time-consuming.
As this is surely not an over-the-top functionality, I assume
there must be a simpler and more efficient way of selecting a
given period within each year, repetitively over several years.
Can someone kindly help me out ?
If you want to treat the March-April data for several years as if
they were a time series (i.e. ignore the gaps), you can give a
special periodicity with the "setobs" command, for example:
series month = $obsminor
smpl month==3 || month==4 --restrict
setobs 61 2004.1 --time-series
This will give you a "calendar" with 61 days per year.
Allin Cottrell
6:07 p.m.
On Thu, 10 Dec 2015, Marc O'Callaghan wrote:
In a time series which comprises daily entries over 10 years, how
may I select a certain period of each year, i.e. only March-April
for instance ?
[...]
I then proceeded to restrict based on the criterion
month >= 2 && month <= 4
[...]
Ah, I see that while my answer at
http://lists.wfu.edu/pipermail/gretl-users/2015-December/011398.html
works for the March-April selection, it doesn't work for the second
variant, February-March-April, since that does not have a fixed
number of observations per year. In the latter case you'd have to
drop back to a custom frequency-1 time series
setobs 1 1 --special-time-series
after sub-sampling on the months in question, to turn the data into
what gretl takes to be a time series.
As this is surely not an over-the-top functionality [...]
Well, it's extremely unusual! Selecting only certain days in the
week to form a time series is quite standard -- provided it's the 5
weekdays or the 5 plus Saturday -- but selecting only certain months
in the year is something I've never heard of.
Allin Cottrell
12:28 p.m.
My apologies, I did mean February to April and not March to April, which
does not help.
" [...] works for the March-April selection, it doesn't work for the second
variant, February-March-April, since that does not have a fixed
number of
observations per year. In the latter case you'd have to drop back to a
custom frequency-1 time series
setobs 1 1 --special-time-series
after sub-sampling on the months in question, to turn the data into what
gretl takes to be a time series. "
In this situation, do you mean I should
subsample each year individually,
and then turn that into a time-series ? Having sampled February-April over
my 10-year period, this command has no effect on the structure of the data
set.
You may be interested in knowing I am working on a climatalogy project,
which involves studying recurring meteorological events over several years.
For this I only need the period of year at which said phenomenon generally
occurs, however the data I order (from Meteoswiss, as it were) is structured
in such a way that I cannot request only the same three-month period over
many years - for one year it works all right, but for more they just send
you the whole yearly series.
Thank you very much for your help, I was completely unaware of the $obsminor
function. I am ashamed to say I am quite useless in matters IT.
Marc O'Callaghan
--------------------------------------------------
From: "Allin Cottrell" <cottrell(a)wfu.edu>
Sent: Friday, December 11, 2015 7:07 PM
To: "Gretl list" <gretl-users(a)lists.wfu.edu>
Subject: Re: [Gretl-users] Help regarding time series
On Thu, 10 Dec 2015, Marc O'Callaghan wrote:
> In a time series which comprises daily entries over 10 years, how may I
> select a certain period of each year, i.e. only March-April for instance
> ?
[...]
> I then proceeded to restrict based on the criterion month >= 2 && month
> <= 4
[...]
Ah, I see that while my answer at
http://lists.wfu.edu/pipermail/gretl-users/2015-December/011398.html
works for the March-April selection, it doesn't work for the second
variant, February-March-April, since that does not have a fixed number of
observations per year. In the latter case you'd have to drop back to a
custom frequency-1 time series
setobs 1 1 --special-time-series
after sub-sampling on the months in question, to turn the data into what
gretl takes to be a time series.
> As this is surely not an over-the-top functionality [...]
Well, it's extremely unusual! Selecting only certain days in the week to
form a time series is quite standard -- provided it's the 5 weekdays or
the 5 plus Saturday -- but selecting only certain months in the year is
something I've never heard of.
Allin Cottrell
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
12:37 p.m.
Am 12.12.2015 um 13:28 schrieb Marc O'Callaghan:
You may be interested in knowing I am working on a climatalogy
project,
which involves studying recurring meteorological events over several
years. For this I only need the period of year at which said phenomenon
generally occurs,
I'm not sure that you really want to "subsample" the data in the gretl
meaning of the term then. If you want to compute stuff only with the
data in those months (but for every year in the same way), maybe you
just need to set the observations in the other months to missing ("NA").
That would give you time series in the standard sense (equi-distant
observations in principle), but with your manually constructed
observation gaps.
Thank you very much for your help, I was completely unaware of the
$obsminor function. I am ashamed to say I am quite useless in matters IT.
No no, the $obsminor is a pretty specialized thing in gretl, and I guess
quite many people do not know it exists.
hth,
sven
11:03 p.m.
On Sat, 12 Dec 2015, Marc O'Callaghan wrote:
My apologies, I did mean February to April and not March to April,
which does
not help.
" [...] works for the March-April selection, it doesn't work for the second
> variant, February-March-April, since that does not have a fixed number of
> observations per year. In the latter case you'd have to drop back to a
> custom frequency-1 time series
>
> setobs 1 1 --special-time-series
>
> after sub-sampling on the months in question, to turn the data
> into what gretl takes to be a time series. "
[...]
You may be interested in knowing I am working on a climatalogy
project, which involves studying recurring meteorological events
over several years. For this I only need the period of year at
which said phenomenon generally occurs, however the data I order
(from Meteoswiss, as it were) is structured in such a way that I
cannot request only the same three-month period over many years -
for one year it works all right, but for more they just send you
the whole yearly series.
Ah, now it's clear where you're coming from -- and you've got me
interested!
I think you best policy, in outline, is:
1) Set up a full-year daily calendar for your data.
2) Import the data using gretl's "join" command. (This assumes the
data are available in comma-separated or another delimited text
format.)
3) Subsample on 90 days per year starting on each February 1. (I
don't suppose you care whether day 90 happens to be called April 31
or May 1.) Then save the reduced dataset.
If you do the above (and therefore end up with a fixed number of
days per year) then you have the choice of treating the data as (a
rather special) time series or as a panel, which might be preferable
for some sorts of analysis.
Here's a toy example using daily maximum temperature data recorded
in Leuchars in Scotland, 2013-2015, courtesy of NOAA. The data file
looks like the following (temperature is in tenths of degrees
Celsius):
date,TMAX
20130101,73
20130105,105
20130108,115
...
It's 7-day per week data, but with irregular missing values. I start
by creating a suitable "blank" dataset:
nulldata 1073 # sufficient number of observations
setobs 7 2013-01-01 # 7-day daily
then do a "join":
join leuchars.csv TMAX --tkey=",%Y%m%d"
To understand the details of "join" you'll have to read the chapter
on "Joining data sources" in the Gretl User's Guide, but basically
this says to import the TMAX variable, matching rows to gretl's
calendar using the first ("date") column, interpreted as YYYYMMDD.
Now, here's the hansl code to get down to 90 days per year:
<hansl>
series year = $obsmajor
series month = $obsminor
series day = $obsmicro
# days in May to January, inclusive
scalar skip = 6*31 + 3*30
# initialize the restriction series
series trimester = 0
i = 1
loop while i <= $nobs --quiet
if month[i] == 2 && day[i] == 1
# mark 90 days from February 1
loop j=1..90 --quiet
trimester[i] = 1
i++
endloop
# for efficiency, skip forward (handling leap years)
i += skip - (day[i]==2)
else
i++
endif
endloop
# restrict to the wanted "trimester"
smpl trimester > 0 --restrict
</hansl>
We can then impose a special time series interpretation, check that
things have gone OK, and save the reduced dataset:
<hansl>
# periodicity 90, starting in 2013
setobs 90 2013.1 --special-time-series
# yrday should run from 1 to 90
series yrday = $obsminor
# do the data look right?
print year month day yrday TMAX --byobs
# if so, save in native format
store trimester.gdt
</hansl>
To treat the dataset as a panel, if you wish:
setobs 90 1.1 --stacked-time-series
Allin Cottrell
8:20 p.m.
Many thanks for your explanations !
Problem solved.
Best,
Marc
--------------------------------------------------
From: "Allin Cottrell" <cottrell(a)wfu.edu>
Sent: Sunday, December 13, 2015 12:03 AM
To: "Gretl list" <gretl-users(a)lists.wfu.edu>
Subject: Re: [Gretl-users] Help regarding time series
On Sat, 12 Dec 2015, Marc O'Callaghan wrote:
> My apologies, I did mean February to April and not March to April, which
> does not help.
>
> " [...] works for the March-April selection, it doesn't work for the
> second
>> variant, February-March-April, since that does not have a fixed number
>> of observations per year. In the latter case you'd have to drop back to
>> a custom frequency-1 time series
>>
>> setobs 1 1 --special-time-series
>>
>> after sub-sampling on the months in question, to turn the data into what
>> gretl takes to be a time series. "
[...]
> You may be interested in knowing I am working on a climatalogy project,
> which involves studying recurring meteorological events over several
> years. For this I only need the period of year at which said phenomenon
> generally occurs, however the data I order (from Meteoswiss, as it were)
> is structured in such a way that I cannot request only the same
> three-month period over many years - for one year it works all right, but
> for more they just send you the whole yearly series.
Ah, now it's clear where you're coming from -- and you've got me
interested!
I think you best policy, in outline, is:
1) Set up a full-year daily calendar for your data.
2) Import the data using gretl's "join" command. (This assumes the data
are available in comma-separated or another delimited text format.)
3) Subsample on 90 days per year starting on each February 1. (I don't
suppose you care whether day 90 happens to be called April 31 or May 1.)
Then save the reduced dataset.
If you do the above (and therefore end up with a fixed number of days per
year) then you have the choice of treating the data as (a rather special)
time series or as a panel, which might be preferable for some sorts of
analysis.
Here's a toy example using daily maximum temperature data recorded in
Leuchars in Scotland, 2013-2015, courtesy of NOAA. The data file looks
like the following (temperature is in tenths of degrees Celsius):
date,TMAX
20130101,73
20130105,105
20130108,115
...
It's 7-day per week data, but with irregular missing values. I start
by creating a suitable "blank" dataset:
nulldata 1073 # sufficient number of observations
setobs 7 2013-01-01 # 7-day daily
then do a "join":
join leuchars.csv TMAX --tkey=",%Y%m%d"
To understand the details of "join" you'll have to read the chapter on
"Joining data sources" in the Gretl User's Guide, but basically this says
to import the TMAX variable, matching rows to gretl's calendar using the
first ("date") column, interpreted as YYYYMMDD.
Now, here's the hansl code to get down to 90 days per year:
<hansl>
series year = $obsmajor
series month = $obsminor
series day = $obsmicro
# days in May to January, inclusive
scalar skip = 6*31 + 3*30
# initialize the restriction series
series trimester = 0
i = 1
loop while i <= $nobs --quiet
if month[i] == 2 && day[i] == 1
# mark 90 days from February 1
loop j=1..90 --quiet
trimester[i] = 1
i++
endloop
# for efficiency, skip forward (handling leap years)
i += skip - (day[i]==2)
else
i++
endif
endloop
# restrict to the wanted "trimester"
smpl trimester > 0 --restrict
</hansl>
We can then impose a special time series interpretation, check that things
have gone OK, and save the reduced dataset:
<hansl>
# periodicity 90, starting in 2013
setobs 90 2013.1 --special-time-series
# yrday should run from 1 to 90
series yrday = $obsminor
# do the data look right?
print year month day yrday TMAX --byobs
# if so, save in native format
store trimester.gdt
</hansl>
To treat the dataset as a panel, if you wish:
setobs 90 1.1 --stacked-time-series
Allin Cottrell
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
3053
days inactive
3060
days old
6 comments
3 participants
participants (3)
-
Allin Cottrell -
Marc O'Callaghan -
Sven Schreiber