5:05 p.m.
Hi,
a colleague made me notice the stack() function for handling a specific
case of panel data import in section 4.5 of the guide. I must admit I
have never been aware of that function, and I have one or two questions
here.
First, it does not appear in the function index (Gretl Command Reference
/ Functions proper, or in the built-in function documentation). Is this
an oversight, or is there a deeper reason?
Secondly, it is well documented in the guide section 4.5, but it appears
to be a strange beast: It is not a gretl command (gets function
arguments in parentheses, for example), but there are double-dash
options such as --offset or --length. I don't remember to have seen
something like this in gretl (or hansl) before.
I guess the story here is some path dependence of the early days, but I
wonder if this area could be cleaned up somehow?
thanks,
sven
11:44 p.m.
On Wed, 5 Sep 2018, Sven Schreiber wrote:
A colleague made me notice the stack() function for handling a
specific case of panel data import in section 4.5 of the guide. I
must admit I have never been aware of that function, and I have
one or two questions here.
First, it does not appear in the function index (Gretl Command
Reference / Functions proper, or in the built-in function
documentation). Is this an oversight, or is there a deeper reason?
Secondly, it is well documented in the guide section 4.5, but it
appears to be a strange beast: It is not a gretl command (gets
function arguments in parentheses, for example), but there are
double-dash options such as --offset or --length. I don't remember
to have seen something like this in gretl (or hansl) before.
I guess the story here is some path dependence of the early days,
but I wonder if this area could be cleaned up somehow?
Sorry it has taken a while for me to get back about this.
Yes, the stack "function" is an oddball. But it's not quite unique
if you look at it the right way. Most gretl commands (if they take
arguments at all) take simple, unstructured or minimally structured
arguments such as series, or lists, or integer lag orders, or
filenames. But in some cases we have a need for more structured
arguments, and then we have defined special function-like thingies
that are command-specific. Examples include the GMM() apparatus for
the "arbond" and "dpanel" commands, and the mds()/mdsl() apparatus
for the "midasreg" command.
But (I hear you say) "stack()" appears in its own right, not as
ancillary to a command. True, and that's the anomaly. My
interpretation is that stack() is ancillary to a silent, unstated
command which we might call "panelize": a means of transforming a
dataset that has been read from CSV or similar, and that in fact
represents a panel, into something that gretl recognizes as panel
data.
Why isn't stack() documented as a function? Because it's not really
a function as understood in modern hansl. It responds to option
flags, like a command. Moreover, it modifies the dimensions of the
dataset, which by modern hansl convention is forbidden for functions
but acceptable for commands (e.g. the "dataset" command).
So what should we do? We could make the implicit "panelize" command
explicit -- but to avoid breaking old scripts we should presumably
continue to accept stack() as documented at present. However, this
whole apparatus, IIRC, pre-dates easy traffic between series and
matrices, and it seems to me that a modern approach to panelization
of an imported dataset in gretl would probably go via matrices
rather than using stack().
Allin
12:30 a.m.
In my reply, at
http://lists.wfu.edu/pipermail/gretl-devel/2018-September/009010.html
to Sven's post on the kinda weird stack() "function" at
http://lists.wfu.edu/pipermail/gretl-devel/2018-September/009009.html
I finished by saying,
So what should we do? We could make the implicit "panelize"
command explicit -- but to avoid breaking old scripts we should
presumably continue to accept stack() as documented at present.
However, this whole apparatus, IIRC, pre-dates easy traffic
between series and matrices, and it seems to me that a modern
approach to panelization of an imported dataset in gretl would
probably go via matrices rather than using stack().
And I thereby left the business unresolved! Here's an attempt at
resolution. It turns out that the problem for which stack() provides
a solution is pretty easily handled by
series -> matrix -> reshape matrix -> series
The problem is described in section 4.5 of the Gretl User's Guide.
That is, we import some panel data that are organized "by variable":
T columns hold the time periods and N rows the "units" or
individuals. More than one variable may be given in such a file, in
which case we assume the k variables are stacked vertically so the
data block as a whole has N*k rows and T columns. How do we make a
gretl panel dataset out of this?
The matrix solution goes as follows, where I assume the "by
variable" data are in foo.csv (with no extraneous rows) and there
are N=10 individuals. (The value of N is the only "extra"
information, not present as such in foo.csv, that is needed.)
<hansl>
open foo.csv
N = 10 # number of individuals
set skip_missing off
matrix X = {dataset}
set skip_missing on
T = cols(X) # number of periods
NT = N * T # total panel series length
k = nelem(X)/NT # "real" number of series
X = mshape(X', NT, k)
nulldata NT --preserve
setobs T 1:1 --stacked-time-series
list Panelvars = null
loop i=1..k --quiet
Panelvars += genseries(sprintf("x%d", i), X[,i])
endloop
store panelized.gdt Panelvars
</hansl>
So here's a suggestion: in section 4.5 of the Guide I delete the
discussion of the fix using the mysterious stack() in favour of an
example and explanation of the method illustrated above.
Should I do that? If so, stack would then be undocumented, but we
could keep it for the time being for backward compatibility.
Allin
7:24 a.m.
Am 10.09.2018 um 02:30 schrieb Allin Cottrell:
And I thereby left the business unresolved! Here's an attempt at
resolution. It turns out that the problem for which stack() provides a
solution is pretty easily handled by
series -> matrix -> reshape matrix -> series
...
So here's a suggestion: in section 4.5 of the Guide I delete the
discussion of the fix using the mysterious stack() in favour of an
example and explanation of the method illustrated above.
Should I do that? If so, stack would then be undocumented, but we
could keep it for the time being for backward compatibility.
Thanks for the suggestion. What I like about it is to clean up the gretl
syntax and remove that "oddball", as you called it. However, replacing a
built-in command/function/whatever by a description of a relatively more
advanced scripting solution is not optimal IMO.
First I thought the functionality of stack() (and of your solution)
could be merged into the 'dataset' or the 'setobs' command -- but
actually the description in the guide ch.4 and the whole business of
stack() is to read in data from external files. So wouldn't 'join' be a
natural candidate for this stuff?
So from a user's point of view the first stack() example in section 4.5
would become something like this:
join panel.txt x1 --panelstack --length=50
join panel.txt x2 --panelstack --length=50 --offset=50
(Perhaps a --no-header option might be needed, but OTOH it might be
implicit in --panelstack.)
I agree that stack() should be kept for a while. For me personally it
was undocumented BTW, because I rely heavily on the built-in command and
function references, where it doesn't appear. Hence the "mysterious" in
this thread's title.
thanks,
sven
12:41 p.m.
On Mon, 10 Sep 2018, Sven Schreiber wrote:
Am 10.09.2018 um 02:30 schrieb Allin Cottrell:
>
>
> And I thereby left the business unresolved! Here's an attempt at
> resolution. It turns out that the problem for which stack() provides a
> solution is pretty easily handled by
>
> series -> matrix -> reshape matrix -> series
>
> ...
>
> So here's a suggestion: in section 4.5 of the Guide I delete the discussion
> of the fix using the mysterious stack() in favour of an example and
> explanation of the method illustrated above.
>
> Should I do that? If so, stack would then be undocumented, but we could
> keep it for the time being for backward compatibility.
Thanks for the suggestion. What I like about it is to clean up the gretl
syntax and remove that "oddball", as you called it. However, replacing a
built-in command/function/whatever by a description of a relatively more
advanced scripting solution is not optimal IMO.
First I thought the functionality of stack() (and of your solution) could be
merged into the 'dataset' or the 'setobs' command -- but actually the
description in the guide ch.4 and the whole business of stack() is to read in
data from external files. So wouldn't 'join' be a natural candidate for this
stuff?
So from a user's point of view the first stack() example in section 4.5 would
become something like this:
join panel.txt x1 --panelstack --length=50
join panel.txt x2 --panelstack --length=50 --offset=50
(Perhaps a --no-header option might be needed, but OTOH it might be implicit
in --panelstack.)
I agree it would be nice from the user's PoV if "join" could handle
this case. However, my first reaction is that this sort of
transposition and reshuffling is not in join's present repertoire and
would induce a fair amount more complexity, plus it means adding at
least three more options to a command that already has more than
any other.
Just thinking out loud, but maybe we could have a pstack() function (a
proper function) that does the business of transposing and reshaping
the data, with "skip_missing" temporarily turned off. It could take
two arguments: the integer number of individuals and an optional list
argument (defaulting to the entire dataset), and would return a matrix
with panelized variables in the columns.
In itself that would leave the user with the task of extracting the
series from the matrix. But maybe this pstack, or whatever we call it,
could write out the transformed dataset as CSV (or gdt) so it could
then be loaded easily -- either in addition to, or instead of,
returning a matrix.
Allin
1:08 p.m.
On Mon, 10 Sep 2018, Allin Cottrell wrote:
I agree it would be nice from the user's PoV if "join"
could handle this
case. However, my first reaction is that this sort of transposition and
reshuffling is not in join's present repertoire and would induce a fair
amount more complexity, plus it means adding at least three more options to a
command that already has more than any other.
I agree on this.
Just thinking out loud, but maybe we could have a pstack() function
(a proper
function) that does the business of transposing and reshaping the data, with
"skip_missing" temporarily turned off. It could take two arguments: the
integer number of individuals and an optional list argument (defaulting to
the entire dataset), and would return a matrix with panelized variables in
the columns.
In itself that would leave the user with the task of extracting the series
from the matrix. But maybe this pstack, or whatever we call it, could write
out the transformed dataset as CSV (or gdt) so it could then be loaded easily
-- either in addition to, or instead of, returning a matrix.
Thi doesn't have to be in libgretl. In fact, it could be part of the
"extra" package, unless I'm missing something.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
1:18 p.m.
On Mon, 10 Sep 2018, Riccardo (Jack) Lucchetti wrote:
On Mon, 10 Sep 2018, Allin Cottrell wrote:
> I agree it would be nice from the user's PoV if "join" could handle
this
> case. However, my first reaction is that this sort of transposition and
> reshuffling is not in join's present repertoire and would induce a fair
> amount more complexity, plus it means adding at least three more options to
> a command that already has more than any other.
I agree on this.
> Just thinking out loud, but maybe we could have a pstack() function (a
> proper function) that does the business of transposing and reshaping the
> data, with "skip_missing" temporarily turned off. It could take two
> arguments: the integer number of individuals and an optional list argument
> (defaulting to the entire dataset), and would return a matrix with
> panelized variables in the columns.
>
> In itself that would leave the user with the task of extracting the series
> from the matrix. But maybe this pstack, or whatever we call it, could write
> out the transformed dataset as CSV (or gdt) so it could then be loaded
> easily -- either in addition to, or instead of, returning a matrix.
Thi doesn't have to be in libgretl. In fact, it could be part of the
"extra"
package, unless I'm missing something.
You're right. Here's proof-of-concept hansl prototype:
<hansl>
function scalar csv_print_matrix (const matrix X,
const string fname,
const strings vnames[null])
N = rows(X)
k = cols(X)
have_names = nelem(vnames) > 0
string nl = sprintf("\n")
outfile @fname --quiet
loop j=1..k -q
if have_names
printf "%s", vnames[j]
else
printf "v%d", j
endif
printf "%s", j == k ? nl : ", "
endloop
loop i=1..N -q
loop j=1..k -q
printf "%.15g", X[i,j]
printf "%s", j == k ? nl : ", "
endloop
endloop
end outfile
return 0
end function
function matrix pstack (int N, list L,
const string fname[null],
const strings vnames[null])
if !exists("vnames")
strings vnames = array(0)
endif
set skip_missing off
matrix X = {L}
set skip_missing on
T = cols(X) # number of periods
NT = N * T # panelized series length
k = nelem(X)/NT # number of panelized series
X = mshape(X', NT, k)
if exists("fname")
csv_print_matrix(X, fname, vnames)
endif
return X
end function
</hansl>
Here's a sample call:
open byvar.csv
N = 6 # number of individuals, A-F
strings vnames = defarray("foo", "bar")
matrix X = pstack(N, dataset, "foo.csv", vnames)
open foo.csv
print -o
where byvar.csv looks like this:
A,1.0,2.0,3.0,4.0
B,1.1,2.1,3.1,4.1
C,1.2,2.2,3.2,4.2
D,1.3,2.3,3.3,4.3
E,1.4,2.4,3.4,4.4
F,1.5,2.5,3.5,4.5
A,10.0,20.0,30.0,40.0
B,10.1,20.1,30.1,40.1
C,10.2,20.2,30.2,40.2
D,10.3,20.3,30.3,40.3
E,10.4,20.4,30.4,40.4
F,10.5,20.5,30.5,40.5
Allin
3:05 p.m.
Am 10.09.2018 um 15:18 schrieb Allin Cottrell:
On Mon, 10 Sep 2018, Riccardo (Jack) Lucchetti wrote:
> On Mon, 10 Sep 2018, Allin Cottrell wrote:
>
>> I agree it would be nice from the user's PoV if "join" could handle
>> this case. However, my first reaction is that this sort of
>> transposition and reshuffling is not in join's present repertoire
>> and would induce a fair amount more complexity, plus it means adding
>> at least three more options to a command that already has more than
>> any other.
>
> I agree on this.
OK.
>>
>> In itself that would leave the user with the task of extracting the
>> series from the matrix. But maybe this pstack, or whatever we call
>> it, could write out the transformed dataset as CSV (or gdt) so it
>> could then be loaded easily -- either in addition to, or instead of,
>> returning a matrix.
>
> Thi doesn't have to be in libgretl. In fact, it could be part of the
> "extra" package, unless I'm missing something.
You're right. Here's proof-of-concept hansl prototype:
OK, as long as 'extra' is shipped with gretl (which it now is, IIRC) I
think it would not be more difficult for the user to call this new
function instead of applying the current solution.
function matrix pstack (int N, list L,
const string fname[null],
const strings vnames[null])
I guess I still have some
thoughts and comments about the function
signature and defaults and what kind of text file input arrangement is
supported in detail, but I don't have time right now. Since it's pure
hansl I'm happy to make concrete code proposals later.
thanks,
sven
7:16 p.m.
On Mon, 10 Sep 2018, Sven Schreiber wrote:
Am 10.09.2018 um 15:18 schrieb Allin Cottrell:
> On Mon, 10 Sep 2018, Riccardo (Jack) Lucchetti wrote:
>>
>> Thi doesn't have to be in libgretl. In fact, it could be part of the
>> "extra" package, unless I'm missing something.
>
> You're right. Here's proof-of-concept hansl prototype:
[of a couple of functions to replace stack()]
OK, as long as 'extra' is shipped with gretl (which it now
is, IIRC) I think
it would not be more difficult for the user to call this new function instead
of applying the current solution.
In fact "extra" is not currently shipped with gretl. That could be
easily arranged, but in that case I think it would be best to put it
into git and treat it as an "addon". That way, as with the other
addons, it would always be up to date in snapshots, and prior versions
would be retained whenever the gretl version requirement is bumped up.
Any thoughts on that?
> function matrix pstack (int N, list L,
> const string fname[null],
> const strings vnames[null])
I guess I still have some thoughts and comments about the function signature
and defaults and what kind of text file input arrangement is supported in
detail, but I don't have time right now. Since it's pure hansl I'm happy to
make concrete code proposals later.
Fine. I now have an improved prototype which I'm happy to supply on
request. It now has some proper error-checking, and it adds "unit" and
"time" series to the CSV file to make it easier to get gretl to
recognize the data as a panel.
Allin
7:59 p.m.
Am 10.09.2018 um 21:16 schrieb Allin Cottrell:
On Mon, 10 Sep 2018, Sven Schreiber wrote:
> OK, as long as 'extra' is shipped with gretl (which it now is, IIRC)
> I think it would not be more difficult for the user to call this new
> function instead of applying the current solution.
In fact "extra" is not currently shipped with gretl. That could be
easily arranged, but in that case I think it would be best to put it
into git and treat it as an "addon". That way, as with the other
addons, it would always be up to date in snapshots, and prior versions
would be retained whenever the gretl version requirement is bumped up.
Any thoughts on that?
I thought we discussed this at some point not so long ago, but maybe
without reaching a firm decision. Also I cannot find any traces of the
discussion, so maybe I'm mixing things up. But in principle it seems
logical I guess. Any disadvantages involved?
thanks,
sven
8:07 p.m.
On Mon, 10 Sep 2018, Sven Schreiber wrote:
Am 10.09.2018 um 21:16 schrieb Allin Cottrell:
> On Mon, 10 Sep 2018, Sven Schreiber wrote:
>
>
>> OK, as long as 'extra' is shipped with gretl (which it now is, IIRC)
>> I think it would not be more difficult for the user to call this new
>> function instead of applying the current solution.
>
> In fact "extra" is not currently shipped with gretl. That could be
> easily arranged, but in that case I think it would be best to put it
> into git and treat it as an "addon". That way, as with the other
> addons, it would always be up to date in snapshots, and prior versions
> would be retained whenever the gretl version requirement is bumped up.
> Any thoughts on that?
>
I thought we discussed this at some point not so long ago, but maybe
without reaching a firm decision.
I have a vague memory of that too...
Also I cannot find any traces of the discussion, so maybe I'm
mixing
things up. But in principle it seems logical I guess. Any
disadvantages involved?
I can't really think of any offhand.
Allin
8:52 p.m.
On Mon, 10 Sep 2018, Allin Cottrell wrote:
On Mon, 10 Sep 2018, Sven Schreiber wrote:
I have a vague memory of that too...
> Also I cannot find any traces of the discussion, so maybe I'm mixing
> things up. But in principle it seems logical I guess. Any disadvantages
> involved?
I can't really think of any offhand.
Me neither.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
6:33 a.m.
Am 10.09.2018 um 21:16 schrieb Allin Cottrell:
On Mon, 10 Sep 2018, Sven Schreiber wrote:
I guess I still have some thoughts and comments about the function
signature and defaults and what kind of text file input arrangement is
supported in detail, but I don't have time right now. Since it's pure
hansl I'm happy to make concrete code proposals later.
Fine. I now have an improved prototype which I'm happy to supply on
request. It now has some proper error-checking, and it adds "unit" and
"time" series to the CSV file to make it easier to get gretl to
recognize the data as a panel.
OK, great. On and after Thursday or Friday I should have some time for
this. If you're not working on it until then, you might as well send it now.
thanks,
sven
1:20 p.m.
On Tue, 11 Sep 2018, Sven Schreiber wrote:
Am 10.09.2018 um 21:16 schrieb Allin Cottrell:
> On Mon, 10 Sep 2018, Sven Schreiber wrote:
> I guess I still have some thoughts and comments about the function
> signature and defaults and what kind of text file input arrangement is
> supported in detail, but I don't have time right now. Since it's pure hansl
> I'm happy to make concrete code proposals later.
>
> Fine. I now have an improved prototype which I'm happy to supply on
> request. It now has some proper error-checking, and it adds "unit" and
> "time" series to the CSV file to make it easier to get gretl to recognize
> the data as a panel.
OK, great. On and after Thursday or Friday I should have some time for this.
If you're not working on it until then, you might as well send it now.
Alright. Hansl and toy test file attached.
Allin
12:12 p.m.
Am 11.09.2018 um 15:20 schrieb Allin Cottrell:
On Tue, 11 Sep 2018, Sven Schreiber wrote:
> Am 10.09.2018 um 21:16 schrieb Allin Cottrell:
>> On Mon, 10 Sep 2018, Sven Schreiber wrote:
>> I guess I still have some thoughts and comments about the function
>> signature and defaults and what kind of text file input arrangement
>> is supported in detail, but I don't have time right now. Since it's
>> pure hansl I'm happy to make concrete code proposals later.
>>
>> Fine. I now have an improved prototype which I'm happy to supply on
>> request. It now has some proper error-checking, and it adds "unit"
>> and "time" series to the CSV file to make it easier to get gretl to
>> recognize the data as a panel.
>
> OK, great. On and after Thursday or Friday I should have some time
> for this. If you're not working on it until then, you might as well
> send it now.
Alright. Hansl and toy test file attached.
Hmm, actually from the explanation in the guide I had a different kind
of input file in mind, namely something like the following (also
attached as a standalone file):
x1
1965 1970 1975 1980 1985
AR 100.0 110.5 118.7 131.2 160.4
AZ 100.0 104.3 113.8 120.9 140.6
x2
1965 1970 1975 1980 1985
AR 103.0 112.5 111.7 136.2 168.4
AZ 109.0 100.3 110.8 128.9 141.6
Your function fails on this kind of file. I have started some
string-processing work to handle this, but I'm having doubts -- what are
we really targeting? It would be helpful to have a real-world example.
thanks,
sven
5:42 p.m.
On Thu, 13 Sep 2018, Sven Schreiber wrote:
Am 11.09.2018 um 15:20 schrieb Allin Cottrell:
> On Tue, 11 Sep 2018, Sven Schreiber wrote:
>
>> Am 10.09.2018 um 21:16 schrieb Allin Cottrell:
>>> On Mon, 10 Sep 2018, Sven Schreiber wrote:
>>> I guess I still have some thoughts and comments about the function
>>> signature and defaults and what kind of text file input arrangement is
>>> supported in detail, but I don't have time right now. Since it's pure
>>> hansl I'm happy to make concrete code proposals later.
>>>
>>> Fine. I now have an improved prototype which I'm happy to supply on
>>> request. It now has some proper error-checking, and it adds "unit"
and
>>> "time" series to the CSV file to make it easier to get gretl to
recognize
>>> the data as a panel.
>>
>> OK, great. On and after Thursday or Friday I should have some time for
>> this. If you're not working on it until then, you might as well send it
>> now.
>
> Alright. Hansl and toy test file attached.
Hmm, actually from the explanation in the guide I had a different kind of
input file in mind, namely something like the following (also attached as a
standalone file):
x1
1965 1970 1975 1980 1985
AR 100.0 110.5 118.7 131.2 160.4
AZ 100.0 104.3 113.8 120.9 140.6
x2
1965 1970 1975 1980 1985
AR 103.0 112.5 111.7 136.2 168.4
AZ 109.0 100.3 110.8 128.9 141.6
Your function fails on this kind of file.
Yes, that's expected. The relevant section of the Guide states that
such a file must be edited, to remove the "extraneous" lines, before
processing via stack() -- although that statement is in a footnote and
may not have the prominence it deserves. In my new, experimental
pstack() function I was making the same assumption: the raw datafile
will have been manually pre-processed to leave just the actual data
blocks.
I have started some string-processing work to handle this, but
I'm
having doubts -- what are we really targeting? It would be helpful
to have a real-world example.
I'd be surprised if it's possible to automate the pre-processing with
any useful generality. But as to what we're targeting, I'm attaching
an example from the IMF (see https://www.imf.org/en/Data , under
"IMF Data Mapper"). IIRC, such files can sometimes have extra lines of
metadata above or below the data block.
Allin
4:18 p.m.
Am 13.09.2018 um 19:42 schrieb Allin Cottrell:
On Thu, 13 Sep 2018, Sven Schreiber wrote:
> I have started some string-processing work to handle this, but I'm
> having doubts -- what are we really targeting? It would be helpful to
> have a real-world example.
I'd be surprised if it's possible to automate the pre-processing with
any useful generality. But as to what we're targeting, I'm attaching
an example from the IMF (see https://www.imf.org/en/Data , under
"IMF Data Mapper"). IIRC, such files can sometimes have extra lines of
metadata above or below the data block.
OK, sorry for the delay. I'm attaching an attempt to read in the data
from such a file (as you posted, imf-gdp.csv) and convert it to a gretl
matrix right away. This string processing is probably not very
efficient, but OTOH it wouldn't be done often, so top speed is not
really an issue I guess.
What still has to be done is to shuffle around the various blocks of the
matrix to match the wanted dataset layout, but that shouldn't be so
difficult.
Comments welcome,
sven
midnight
On Wed, 26 Sep 2018, Sven Schreiber wrote:
Am 13.09.2018 um 19:42 schrieb Allin Cottrell:
> On Thu, 13 Sep 2018, Sven Schreiber wrote:
>
>> I have started some string-processing work to handle this, but I'm having
>> doubts -- what are we really targeting? It would be helpful to have a
>> real-world example.
>
> I'd be surprised if it's possible to automate the pre-processing with any
> useful generality. But as to what we're targeting, I'm attaching an example
> from the IMF (see https://www.imf.org/en/Data , under
> "IMF Data Mapper"). IIRC, such files can sometimes have extra lines of
> metadata above or below the data block.
OK, sorry for the delay. I'm attaching an attempt to read in the data from
such a file (as you posted, imf-gdp.csv) and convert it to a gretl matrix
right away. This string processing is probably not very efficient, but OTOH
it wouldn't be done often, so top speed is not really an issue I guess.
What still has to be done is to shuffle around the various blocks of the
matrix to match the wanted dataset layout, but that shouldn't be so
difficult.
Nice work! My only question is: how sure are we that the IMF data
files are consistent in their presentation, to the extent that your
function will work on any example? Well, also: are there other data
sources where panel data are provided in a similar manner (time
series running horizontally) and if so, would they also be amenable
to your approach?
One other thought: IMO the most desirable replacement for the
existing stack() should probably also be able to handle (I suppose,
via some option) the (simpler) case where we have a dataset with one
or more blocks of N time-series of length T, where time goes
vertically but the individuals are arrayed horizontally, in separate
series, and we'd like to stack the N series into one panel series.
For example, we open a CSV file which has N columns holding GDP
1990-2017 for N countries, and we want to panelize GDP. This may be
a "join" task; I haven't thought it through yet, but I suspect that
even if its doable via join there may be an advantage in automating
it via some pstack() variant.
Allin
5:48 a.m.
On Wed, 26 Sep 2018, Allin Cottrell wrote:
One other thought: IMO the most desirable replacement for the
existing
stack() should probably also be able to handle (I suppose, via some option)
the (simpler) case where we have a dataset with one or more blocks of N
time-series of length T, where time goes vertically but the individuals are
arrayed horizontally, in separate series, and we'd like to stack the N series
into one panel series.
For example, we open a CSV file which has N columns holding GDP 1990-2017 for
N countries, and we want to panelize GDP. This may be a "join" task; I
haven't thought it through yet, but I suspect that even if its doable via
join there may be an advantage in automating it via some pstack() variant.
I have some hansl code that does just that. I'll send it to Sven later
today.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
5:54 a.m.
Am 27.09.2018 um 02:00 schrieb Allin Cottrell:
On Wed, 26 Sep 2018, Sven Schreiber wrote:
> OK, sorry for the delay. I'm attaching an attempt to read in the data
> from such a file (as you posted, imf-gdp.csv) and convert it to a
> gretl matrix right away. This string processing is probably not very
> efficient, but OTOH it wouldn't be done often, so top speed is not
> really an issue I guess.
>
> What still has to be done is to shuffle around the various blocks of
> the matrix to match the wanted dataset layout, but that shouldn't be
> so difficult.
Nice work! My only question is: how sure are we that the IMF data
files are consistent in their presentation, to the extent that your
function will work on any example? Well, also: are there other data
sources where panel data are provided in a similar manner (time series
running horizontally) and if so, would they also be amenable to your
approach?
The orientation of the data is a matter of the to-be-written second (or
top-level) function. Probably the user would have to specify whether
time is in rows or columns and so on.
This present function just reads in a rectangular data array (matrix)
from a text file, automatically discarding rows that for whatever reason
differ in their length.
(These could be empty rows or descriptive stuff at the bottom, or rows
with just the variable names in between.)
This should even cover missing data (unbalanced panels for example) as
long as the layout is coherent in that missing data (a) either has some
string code or (b) the entries are simply empty but the entry as such is
signalled by the correct amount of column separators. But this hasn't
been tested yet.
So right now I'm somewhat optimistic that it would work.
One other thought: IMO the most desirable replacement for the
existing
stack() should probably also be able to handle (I suppose, via some
option) the (simpler) case where we have a dataset with one or more
blocks of N time-series of length T, where time goes vertically but
the individuals are arrayed horizontally, in separate series, and we'd
like to stack the N series into one panel series.
For example, we open a CSV file which has N columns holding GDP
1990-2017 for N countries, and we want to panelize GDP. This may be a
"join" task; I haven't thought it through yet, but I suspect that even
if its doable via join there may be an advantage in automating it via
some pstack() variant.
If I understand correctly the resulting matrix from my present function
would be TxN in this case, and since gretl's native panel format is
stacked time series (right?) doing vec(M) should then be enough, as in
"series GDP = vec(M)", possibly preceded by "nulldata nelem(M)
--preserve" and "setobs 1 1:1 --stacked-time-series" or something.
cheers,
sven
6:13 a.m.
On Wed, 26 Sep 2018, Sven Schreiber wrote:
>
>> OK, sorry for the delay. I'm attaching an attempt
Minor correction: The included helper function matof (as in: matrix
variant of atof) had a little redundancy, because atof itself already
handles empty strings. Here's the update:
<hansl>
function matrix matof(strings in, int skip[0::0])
# Returns column vector with the numerical values in the string array.
# 'skip' refers to how many leading array elements should be discarded.
N = nelem(in)
matrix m = ones(N - skip,1) * $huge # initialize to nonsense
loop i = 1+skip .. N --quiet
m[i - skip] = atof(in[i])
endloop
return m
end function
</hansl>
cheers,
sven
6:27 a.m.
On Thu, 27 Sep 2018, Sven Schreiber wrote:
On Wed, 26 Sep 2018, Sven Schreiber wrote:
>>
>>> OK, sorry for the delay. I'm attaching an attempt
Minor correction: The included helper function matof (as in: matrix variant
of atof) had a little redundancy, because atof itself already handles empty
strings. Here's the update:
<hansl>
function matrix matof(strings in, int skip[0::0])
# Returns column vector with the numerical values in the string array.
# 'skip' refers to how many leading array elements should be discarded.
N = nelem(in)
matrix m = ones(N - skip,1) * $huge # initialize to nonsense
loop i = 1+skip .. N --quiet
m[i - skip] = atof(in[i])
endloop
return m
end function
</hansl>
I was wondering, yesterday, when looking at your code: why initialise m to
$huge? wouldn't
matrix m = mshape(NA, N - skip, 1)
do?
And another thing: it may make sense to expand atof() so as to return
automatically a column vector if the argument is a string array. Or am I
missing something?
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
6:32 a.m.
Am 27.09.2018 um 08:27 schrieb Riccardo (Jack) Lucchetti:
On Thu, 27 Sep 2018, Sven Schreiber wrote:
I was wondering, yesterday, when looking at your code: why initialise
m to $huge? wouldn't
matrix m = mshape(NA, N - skip, 1)
do?
Absolutely, I just wasn't aware that mshape works with NA as input.
And another thing: it may make sense to expand atof() so as to return
automatically a column vector if the argument is a string array. Or am
I missing something?
That would be nice, yes.
Speaking about the helper functions, what do you think of
<hansl>
function matrix mode(matrix v)
# v should be a vector
E = ecdf(vec(v))
howmuch = diff(E[,2])
where = imaxc(howmuch)
return E[where, 1] | howmuch[where]
end function
</hansl>
I must say I quite like it. Should it go into the extra addon?
-sven
9:33 a.m.
On Thu, 27 Sep 2018, Sven Schreiber wrote:
Speaking about the helper functions, what do you think of
<hansl>
function matrix mode(matrix v)
# v should be a vector
E = ecdf(vec(v))
howmuch = diff(E[,2])
where = imaxc(howmuch)
return E[where, 1] | howmuch[where]
end function
</hansl>
I must say I quite like it. Should it go into the extra addon?
I was just thinking about it. I vaguely remember a debate we had on one of
the two lists at some point (but I may be wrong, I tried googling for it,
to no avail) about implementing a mode() function and we got collectively
stuck on the case when you have multimodal data. Plus, diff-ing the output
of ecdf() wull give you trouble if the mode happens to be the smallest
element of the vector.
For example, consider this (I'm using Sven's function above)
<hansl>
matrix uno = seq(1,3)' ** ones(3,1)
matrix due = seq(1,3)' ** {1;0;0}
print uno due
eval mode(uno)
eval mode(due)
</hansl>
As you can see if you run it, mode(uno) is debatable, mode(due) is plain
wrong.
A remedy for the second case may be something like
<hansl>
function matrix mode(matrix v)
vv = values(v)
A = sumc(vec(v) .= vv')
where = imaxr(A)
return vv[where] | A[where]/nelem(v)
end function
</hansl>
but then, in case of multi-modal data, what should be done is debatable.
I see you're using the mode function as a heuristic criterion to find the
data dimensions; maybe we could figure out something else.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
11:09 a.m.
Am 27.09.2018 um 11:33 schrieb Riccardo (Jack) Lucchetti:
I was just thinking about it. I vaguely remember a debate we had on
one of the two lists at some point (but I may be wrong, I tried
googling for it, to no avail) about implementing a mode() function and
we got collectively stuck on the case when you have multimodal data.
Plus, diff-ing the output of ecdf() wull give you trouble if the mode
happens to be the smallest element of the vector.
True, that was a bug. Try this corrected version:
<hansl>
function matrix mode(matrix v)
# v should be a vector
E = ecdf(vec(v))
howmuch = diff(0 | E[,2])[2:] # make sure the 1st is also diffed
where = imaxc(howmuch)
return E[where, 1] | howmuch[where]
end function
</hansl>
but then, in case of multi-modal data, what should be done is debatable.
This is partly inherited from the imax*/imin* suite of gretl functions.
In 2015 we briefly discussed this on-list. I wrote:
' FWIW, (Python's) Numpy's argmax() and argmin() functions explicitly
note that:
"In case of multiple occurrences of the maximum values, the indices
corresponding to the first occurrence are returned."'
And you answered:
'This makes sense. I'm not sure if in fact we follow this policy, but if
we agree we should I can have a go the the C code to make sure we do,
and of course update the docs.'
It looks as if the gretl docs are still silent on the issue. Actually I
was surprised that apparently imaxc in your example uno gave the last
value, I thought it would return the first of the multi-modes.
I see you're using the mode function as a heuristic criterion to
find
the data dimensions; maybe we could figure out something else.
I'm happy to hear suggestions. The max line length didn't work because
of the file structure.
thanks,
sven
6:58 a.m.
Am 27.09.2018 um 07:54 schrieb Sven Schreiber:
The orientation of the data is a matter of the to-be-written second
(or top-level) function. Probably the user would have to specify
whether time is in rows or columns and so on.
Here (attached) is a another iteration with some bugfixes, but
especially the mentioned second function (pstack_sven) which transforms
everything to a gretl series.
>
> For example, we open a CSV file which has N columns holding GDP
> 1990-2017 for N countries, and we want to panelize GDP.
In theory this is now handled as well by specifying timevertical=1 in
pstack_sven. But untested.
Note I'm working under the assumption that internally in gretl the panel
data organization is always as stacked time series, no matter whether
setobs gave the --stacked-cross-section option or not. I hope this is
correct, otherwise please speak up.
cheers,
sven
1:42 p.m.
New subject: atof for many inputs (was Re: The mysterious stack() function)
Am 27.09.2018 um 08:32 schrieb Sven Schreiber:
Am 27.09.2018 um 08:27 schrieb Riccardo (Jack) Lucchetti:
>
> And another thing: it may make sense to expand atof() so as to return
> automatically a column vector if the argument is a string array. Or
> am I missing something?
>
That would be nice, yes.
Is this still on the table? (To extend the coverage of 'atof' in core
gretl.)
If not, the matof() function of this thread a while ago would also be a
candidate for inclusion into extra.
Thanks,
sven
1:44 p.m.
New subject: atof for many inputs (was Re: The mysterious stack() function)
On Fri, 19 Oct 2018, Sven Schreiber wrote:
Am 27.09.2018 um 08:32 schrieb Sven Schreiber:
> Am 27.09.2018 um 08:27 schrieb Riccardo (Jack) Lucchetti:
>>
>> And another thing: it may make sense to expand atof() so as to return
>> automatically a column vector if the argument is a string array. Or am I
>> missing something?
>>
>
> That would be nice, yes.
Is this still on the table? (To extend the coverage of 'atof' in core gretl.)
If not, the matof() function of this thread a while ago would also be a
candidate for inclusion into extra.
Check out sscanf() first.
Allin
2:13 p.m.
New subject: atof for many inputs (was Re: The mysterious stack() function)
Am 19.10.2018 um 15:44 schrieb Allin Cottrell:
On Fri, 19 Oct 2018, Sven Schreiber wrote:
>
> Is this still on the table? (To extend the coverage of 'atof' in core
> gretl.)
> If not, the matof() function of this thread a while ago would also be
> a candidate for inclusion into extra.
Check out sscanf() first.
Thanks for the hint. What's missing then (AFAICS) is a function that
reverses strplit, that is, joins the members of an array of strings into
a (separator-separated) single string.
Brute force:
function string arr2sepstr(strings S, string sep[null])
if !exists(sep)
string sep = " " # blank as default
endif
string out = ""
loop i=1..nelem(S) -q
out ~= S[i] ~ sep
endloop
# discard trailing separator
return substr(out, 1, nelem(out)-1)
end function
The rest indeed seems to be done by sscanf(arr2sepstr(Sarr), %m, mymatrix).
cheers,
sven
3:09 p.m.
Am 28.09.2018 um 08:58 schrieb Sven Schreiber:
Am 27.09.2018 um 07:54 schrieb Sven Schreiber:
>
> The orientation of the data is a matter of the to-be-written second
> (or top-level) function. Probably the user would have to specify
> whether time is in rows or columns and so on.
Here (attached) is a another iteration with some bugfixes, but
especially the mentioned second function (pstack_sven) which transforms
everything to a gretl series.
>>
>> For example, we open a CSV file which has N columns holding GDP
>> 1990-2017 for N countries, and we want to panelize GDP.
In theory this is now handled as well by specifying timevertical=1 in
pstack_sven. But untested.
Note I'm working under the assumption that internally in gretl the panel
data organization is always as stacked time series, no matter whether
setobs gave the --stacked-cross-section option or not. I hope this is
correct, otherwise please speak up.
Hi,
I almost forgot about this one... I guess we need to put it on the
feature tracker, otherwise this will get lost...
cheers,
sven
12:03 p.m.
Am 31.12.2018 um 16:09 schrieb Sven Schreiber:
Am 28.09.2018 um 08:58 schrieb Sven Schreiber:
> Am 27.09.2018 um 07:54 schrieb Sven Schreiber:
>>
> Here (attached) is a another iteration with some bugfixes, but
> especially the mentioned second function (pstack_sven) which
> transforms everything to a gretl series.
>
Hi,
I almost forgot about this one... I guess we need to put it on the
feature tracker, otherwise this will get lost...
This is now in https://sourceforge.net/p/gretl/feature-requests/115/
cheers,
sven
1925
days inactive
2053
days old
30 comments
3 participants
participants (3)
-
Allin Cottrell -
Riccardo (Jack) Lucchetti -
Sven Schreiber