9:36 a.m.
Dear all,
I am trying to load a bunch of data into gretl, with a structure similar to the attached
files (time 1600), i.e. I have horizontal and vertical appending at the same time. I
basically use a script that opens the first file (like parameters_1.tsv) and then via a
loop appends all the other files. However, the horizontal appending works only for the
first files (belonging to *_1.tsv) and even there not completely, afterwards a single
entry is done for each file, with "missing" values. There is a common key in the
first column, so I thought the "update-overlap" option should work (after the
first record is complete (all 9 files above appended) and the first "item" of
each record is added (see script attached). It does not, however....
PS: the "--quiet" option is not present in the tool-tip of "append".
A WE is given with the script and the 18 files (first 2 records of my simulation
experiment). The operation should lead to two records, but it ends up with 11... Take a
look at the common key "ABMAT_ConfigID" to see what I mean with one glance,
after you imported the data.
Any idea what I am missing?
Best Frederik
9:54 a.m.
Am 19.09.2017 um 11:36 schrieb Schaff, Frederik:
I am trying to load a bunch of data into gretl, with a structure
similar
to the attached files (time 1600), i.e. I have horizontal and vertical
appending at the same time. I basically use a script that opens the
first file (like parameters_1.tsv) and then via a loop appends all the
other files. However, the horizontal appending works only for the first
files (belonging to *_1.tsv) and even there not completely,
OK, this is only a generic answer, sorry: First, I don't exactly
understand what you mean with "horizontal" or "vertical". Second, for
more general dataset handling there's also the 'join' command in
addition to append. Maybe that's what you need.
cheers,
sven
1:24 p.m.
Am 19.09.2017 um 11:36 schrieb Schaff, Frederik:
> I am trying to load a bunch of data into gretl, with a structure
> similar to the attached files (time 1600), i.e. I have horizontal and
> vertical appending at the same time. I basically use a script that
> opens the first file (like parameters_1.tsv) and then via a loop
> appends all the other files. However, the horizontal appending works
> only for the first files (belonging to *_1.tsv) and even there not
> completely,
OK, this is only a generic answer, sorry: First, I don't exactly understand what
you mean with "horizontal" or "vertical". Second, for more general
dataset
handling there's also the 'join' command in addition to append. Maybe
that's
what you need.
cheers,
sven
Ok, I guess the script and files attached are self-explanatory, but I'll try to
explain it without them.
I have a set of files with multiple columns and (currently) two lines - head and data. The
First column always holds a common key, the next columns hold several different metrics.
Horizontal merging means that there are several files that contain information on the same
"record" which is specified by the key in the first column. In other words, the
data belonging to a single record is split in several files.
Vertical appending means that there are several files for different records, also.
The "join" command would, I guess, work. But from what I understand, I need to
explicitly name all the single columns I want to import.
Alternatively, if there was a command / operation to ... clean-up (?) the mess that is
currently generated, that would also help.
The database I get currently looks like the following:
..................................................................................................
: Common Key : file1-data1 : file1-data2 : file2-data3 : file2-data4 : file3-data1 :
file3-data2 :
:............:.............:.............:.............:.............:.............:.............:
: 1 : data : data : : : :
:
: 1 : : : data : data : data : data
:
: 2 : data : data : : : :
:
: 2 : : : data : data : :
:
: 2 : : : : : data : data
:
:............:.............:.............:.............:.............:.............:.............:
1:46 p.m.
Am 19.09.2017 um 15:24 schrieb Schaff, Frederik:
> Am 19.09.2017 um 11:36 schrieb Schaff, Frederik:
>
>> I am trying to load a bunch of data into gretl, with a structure
>> similar to the attached files (time 1600), i.e. I have horizontal and
>> vertical appending at the same time.
Ok, I guess the script and files attached are self-explanatory, but
I'll try to explain it without them.
Yes, sorry, I don't have time right now to look at your files in detail.
According to the help for 'append', your horizontal/vertical thing is
supported in principle. ("The appended data may take the form of either
additional observations on series already present in the dataset, and/or
new series.") AFAICS, "new series" is equivalent to your
"horizontal",
and "additional obs" == "vertical".
Note the condition given in the help: "In the case of adding series,
compatibility requires either (a) that the number of observations for
the new data equals that for the current data, or (b) that the new data
carries clear observation information so that gretl can work out how to
place the values."
Is that the case for your files?
Also, using an explicit 'setobs' command after loading the first dataset
might possibly help.
good luck,
sven
2:36 p.m.
Am 19.09.2017 um 15:24 schrieb Schaff, Frederik:
>
>> Am 19.09.2017 um 11:36 schrieb Schaff, Frederik:
>>
>>> I am trying to load a bunch of data into gretl, with a structure
>>> similar to the attached files (time 1600), i.e. I have horizontal
>>> and vertical appending at the same time.
> Ok, I guess the script and files attached are self-explanatory, but I'll try to
explain it without them.
Yes, sorry, I don't have time right now to look at your files in detail.
According to the help for 'append', your horizontal/vertical thing is supported
in principle. ("The appended data may take the form of either additional
observations on series already present in the dataset, and/or new series.")
AFAICS, "new series" is equivalent to your "horizontal", and
"additional obs"
== "vertical".
Note the condition given in the help: "In the case of adding series,
compatibility requires either (a) that the number of observations for the new
data equals that for the current data, or (b) that the new data carries clear
observation information so that gretl can work out how to place the values."
Is that the case for your files?
Also, using an explicit 'setobs' command after loading the first dataset might
possibly help.
good luck,
sven
Setobs does not help. I found a small typo, but that effects only the "first"
anomaly (one series was added twice, why gretl switched from horicontal to vertical
appending). However, this may hint to the internal procedure. I guess gretl has a problem
with adding observations record in pieces. I.e., consider the dataset as rows with data
(+) or missing values (0) separated by |. First, gretl correctly appends to the right
(increasing the items from the given observation record). But when I "move" to
the next row, gretl directly fills missing values with n/a and then, in the next steps,
does not fill in the missing values but produces a new observation record with lots of
missing values, each time:
open a1 -> +|
append a2 -> ++|
append a3 -> +++|
works, but then it becomes
append b1 -> +++|+00
append b2 -> +++|+00|0+0
append b3 -> +++|+00|0+0|00+
instead of
append b1 -> +++|+
append b2 -> +++|++
append b3 -> +++|+++
Any idea how to change that? I thought that starting from the second record the option
" --update-overlap" would help, but it doesn't. Is there a way to explicitly
provide a "key" for append?
2:42 p.m.
Am 19.09.2017 um 16:36 schrieb Schaff, Frederik:
Setobs does not help. I found a small typo, but that effects only the
"first" anomaly (one series was added twice, why gretl switched from horicontal
to vertical appending). However, this may hint to the internal procedure. I guess gretl
has a problem with adding observations record in pieces.
It sounds as if you _think_ you found a bug. If so, it would help to
provide a minimal working script example (not always easy, I know).
Is there a way to explicitly provide a "key" for append?
"key" again tingles my 'join' alert.
If there's a bug in 'append', it would be very useful if you can
pinpoint it. If not, I think 'joint' appears more frightening at first
than it really is.
cheers,
sven
9:22 p.m.
On Tue, 19 Sep 2017, Schaff, Frederik wrote:
I guess gretl has a problem with adding observations record in
pieces. [...]
Yes, that's right. Consider the following miniaturization of what
you're trying to do:
$ cat little1.csv
obs,a,b
1,1,2
$ cat little2.csv
obs,c,d
1,3,4
$cat little3.csv
obs,a,b
2,5,6
$cat little4.csv
obs,c,d
2,7,8
Now we try running this:
<hansl>
open little1.csv -q
print -o
append little2.csv -q
print -o
append little3.csv -q
print -o
append little4.csv -q
print -o
</hansl>
Appending little2 goes OK: adding two columns. Appending little3 is
also OK[*]: adding a row (albeit an incomplete one). But appending
little4 fails. So far as current gretl is concerned this is not an
"append": we're adding neither variables nor observations (in the
sense of rows). Rather it's a "fill in the blanks" operation.
It's arguable that "append" should handle this case but it'll take
some work to get it right.
[*] This statement may actually depend on a little fix that's now in
git.
Allin
6:46 p.m.
On Tue, 19 Sep 2017, Schaff, Frederik wrote:
I am trying to load a bunch of data into gretl, with a structure
similar to the attached files (time 1600), i.e. I have horizontal
and vertical appending at the same time. I basically use a script
that opens the first file (like parameters_1.tsv) and then via a
loop appends all the other files. However, the horizontal appending
works only for the first files (belonging to *_1.tsv) and even there
not completely, afterwards a single entry is done for each file,
with "missing" values. There is a common key in the first column, so
I thought the "update-overlap" option should work [...]
Hold it there! The simple "append" mechanism has no way of defining
(or guessing at) a "key": for that you need the "join" command. As it
is, your "ABMAT_ConfigID" is treated as just a regular variable.
However, I can see that since you want to import hundreds of series,
"join" is not going to be very convenient. In this case I think I'd
recommend using matrices to piece the dataset together. A sample
script follows. (I've fixed what I take to be a bug in your version,
namely setting "path_interval_f" and "path_interval_h" to the same
string and hence appending the same data twice.)
<hansl>
set verbose off
clear
# string path_base = "F:/LSDgit/Work/EFBP_original/Results/Original/Aggregate/"
string path_base = ""
string path_ts = path_base ~ "parameters_"
string ts_name = ".tsv"
scalar first = 1
scalar last = 2 #1600
string fname = ""
strings iname = array(9)
iname[1] = "stat_0-499_"
iname[2] = "stat_500-999_"
iname[3] = "stat_1000-1499_"
iname[4] = "stat_1500-1999_"
iname[5] = "stat_1500-1504_"
iname[6] = "stat_1500-1519_"
iname[7] = "stat_1500-1549_"
iname[8] = "stat_1500-1599_"
iname[9] = "stat_1500-1749_"
matrix X = {}
# Step 1: get all the columns in place
sprintf fname "%s%d%s", path_ts, 1, ts_name
open @fname --preserve --quiet
printf " got %d vars, %d obs\n", $nvars, $nobs
X = {dataset}
strings vnames = varnames(dataset)
loop j=1..9 -q
fname = sprintf("%s%s%d%s", path_base, iname[j], 1, ts_name)
open @fname --preserve --quiet
printf " got %d vars, %d obs\n", $nvars, $nobs
vnames += varnames(dataset)
# add this dataset horizontally
X ~= {dataset}
endloop
# Step 2: append additional rows
loop i=2..last -q
sprintf fname "%s%d%s", path_ts, i, ts_name
open @fname --preserve --quiet
matrix addX = {dataset}
printf " got %d vars, %d obs\n", $nvars, $nobs
loop j=1..9 -q
fname = sprintf("%s%s%d%s", path_base, iname[j], i, ts_name)
open @fname --preserve --quiet
printf " got %d vars, %d obs\n", $nvars, $nobs
addX ~= {dataset}
endloop
printf "addX: %d x %d\n", rows(addX), cols(addX)
# add the whole addX vertically
X |= addX
endloop
colnames(X, vnames)
N = rows(X)
nulldata N --preserve
loop i=1..cols(X)
genseries(colname(X, i), X[,i])
endloop
delete index
varlist
print ABMAT_ConfigID se_Seed sh_Seed MA_Utility_MAE_uq_I9 -o
</hansl>
Allin
6:54 a.m.
Dear Allin,
thank you very much! This works like a charm now.
Best
Frederik
-----Ursprüngliche Nachricht-----
Von: gretl-users-bounces(a)lists.wfu.edu [mailto:gretl-users-
bounces(a)lists.wfu.edu] Im Auftrag von Allin Cottrell
Gesendet: Dienstag, 19. September 2017 20:47
An: Gretl list <gretl-users(a)lists.wfu.edu>
Betreff: Re: [Gretl-users] WG: append horizontal and vertical
On Tue, 19 Sep 2017, Schaff, Frederik wrote:
> I am trying to load a bunch of data into gretl, with a structure
> similar to the attached files (time 1600), i.e. I have horizontal and
> vertical appending at the same time. I basically use a script that
> opens the first file (like parameters_1.tsv) and then via a loop
> appends all the other files. However, the horizontal appending works
> only for the first files (belonging to *_1.tsv) and even there not
> completely, afterwards a single entry is done for each file, with
> "missing" values. There is a common key in the first column, so I
> thought the "update-overlap" option should work [...]
Hold it there! The simple "append" mechanism has no way of defining (or
guessing at) a "key": for that you need the "join" command. As it is,
your
"ABMAT_ConfigID" is treated as just a regular variable.
However, I can see that since you want to import hundreds of series, "join" is
not going to be very convenient. In this case I think I'd recommend using
matrices to piece the dataset together. A sample script follows. (I've fixed
what I take to be a bug in your version, namely setting "path_interval_f" and
"path_interval_h" to the same string and hence appending the same data
twice.)
<hansl>
set verbose off
clear
# string path_base =
"F:/LSDgit/Work/EFBP_original/Results/Original/Aggregate/"
string path_base = ""
string path_ts = path_base ~ "parameters_"
string ts_name = ".tsv"
scalar first = 1
scalar last = 2 #1600
string fname = ""
strings iname = array(9)
iname[1] = "stat_0-499_"
iname[2] = "stat_500-999_"
iname[3] = "stat_1000-1499_"
iname[4] = "stat_1500-1999_"
iname[5] = "stat_1500-1504_"
iname[6] = "stat_1500-1519_"
iname[7] = "stat_1500-1549_"
iname[8] = "stat_1500-1599_"
iname[9] = "stat_1500-1749_"
matrix X = {}
# Step 1: get all the columns in place
sprintf fname "%s%d%s", path_ts, 1, ts_name open @fname --preserve --
quiet printf " got %d vars, %d obs\n", $nvars, $nobs X = {dataset} strings
vnames = varnames(dataset) loop j=1..9 -q
fname = sprintf("%s%s%d%s", path_base, iname[j], 1, ts_name)
open @fname --preserve --quiet
printf " got %d vars, %d obs\n", $nvars, $nobs
vnames += varnames(dataset)
# add this dataset horizontally
X ~= {dataset}
endloop
# Step 2: append additional rows
loop i=2..last -q
sprintf fname "%s%d%s", path_ts, i, ts_name
open @fname --preserve --quiet
matrix addX = {dataset}
printf " got %d vars, %d obs\n", $nvars, $nobs
loop j=1..9 -q
fname = sprintf("%s%s%d%s", path_base, iname[j], i, ts_name)
open @fname --preserve --quiet
printf " got %d vars, %d obs\n", $nvars, $nobs
addX ~= {dataset}
endloop
printf "addX: %d x %d\n", rows(addX), cols(addX)
# add the whole addX vertically
X |= addX
endloop
colnames(X, vnames)
N = rows(X)
nulldata N --preserve
loop i=1..cols(X)
genseries(colname(X, i), X[,i])
endloop
delete index
varlist
print ABMAT_ConfigID se_Seed sh_Seed MA_Utility_MAE_uq_I9 -o
</hansl>
Allin
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
9:51 a.m.
On Tue, 19 Sep 2017, Allin Cottrell wrote:
However, I can see that since you want to import hundreds of series,
"join"
is not going to be very convenient.
Heh. I beg to differ :)
Here's an alternative implementation which uses "join". Evaluating the
relative merits of the two solutions is left as an exercise to the reader.
<hansl>
set verbose off
clear
#string path_base = "E:/RESULTS/ElFarol/Original/"
string path_base = ""
string path_ts = path_base ~ "parameters_"
string ts_name = ".tsv"
scalar first = 1
scalar last = 2 #1600
string temp_path = ""
sprintf temp_path "%s%d%s",path_ts,first,ts_name
open @temp_path --preserve --quiet
loop i=first+1 .. last
fname = sprintf("%s%d%s",path_ts,i,ts_name)
append @fname --quiet
endloop
strings intervals =
defarray("0-499","500-999","1000-1499",\
"1500-1999","1500-1504","1500-1519","1500-1549",\
"1500-1519","1500-1749")
scalar n = nelem(intervals)
loop i=1..n --quiet
fname =
sprintf("%s%s%s%s",path_base,"stat_",intervals[i],"_1.tsv")
content = readfile(fname)
v = ""
getline(content, v)
strings vars = strsplit(v)
v = ""
loop j = 2..nelem(vars) --quiet
v ~= sprintf("%s ",fixname(vars[j]))
endloop
loop j = first .. last --quiet
fname =
sprintf("%s%s%s_%d.tsv",path_base,"stat_",intervals[i], j)
cmd = sprintf(" %s %s --ikey=ABMAT_ConfigID\n", fname, v)
join @cmd
printf "%d, %d\n", i,j
flush
endloop
endloop
varlist
print ABMAT_ConfigID se_Seed sh_Seed MA_Utility_MAE_uq_I9 -o
</hansl>
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
11:55 a.m.
> However, I can see that since you want to import hundreds of
series, "join"
> is not going to be very convenient.
Heh. I beg to differ :)
Here's an alternative implementation which uses "join". Evaluating the
relative merits of the two solutions is left as an exercise to the reader.
Thanks. This is more concise (and in some points more general - the rest I changed), but
the join command does not work...? The problem seems to be that
<hansl>
cmd = sprintf(" %s %s --ikey=ABMAT_ConfigID\n", fname, v)
</hansl>
Does somehow not parse "v" to cmd. If I print cmd, the content of "v"
is missing. If I only print v, it is fine. Is there a maximum string length or such?
Best
Frederik
<hansl>
set verbose off
clear
#string path_base = "E:/RESULTS/ElFarol/Original/"
string path_base = ""
string path_ts = path_base ~ "parameters_"
string ts_name = ".tsv"
scalar first = 1
scalar last = 2 #1600
string temp_path = ""
sprintf temp_path "%s%d%s",path_ts,first,ts_name open @temp_path --
preserve --quiet
loop i=first+1 .. last
fname = sprintf("%s%d%s",path_ts,i,ts_name)
append @fname --quiet
endloop
strings intervals =
defarray("0-499","500-999","1000-1499",\
"1500-1999","1500-1504","1500-1519","1500-1549",\
"1500-1519","1500-1749")
scalar n = nelem(intervals)
loop i=1..n --quiet
fname =
sprintf("%s%s%s%s",path_base,"stat_",intervals[i],"_1.tsv")
content = readfile(fname)
v = ""
getline(content, v)
strings vars = strsplit(v)
v = ""
loop j = 2..nelem(vars) --quiet
v ~= sprintf("%s ",fixname(vars[j]))
endloop
loop j = first .. last --quiet
fname =
sprintf("%s%s%s_%d.tsv",path_base,"stat_",intervals[i], j)
cmd = sprintf(" %s %s --ikey=ABMAT_ConfigID\n", fname, v)
join @cmd
printf "%d, %d\n", i,j
flush
endloop
endloop
varlist
print ABMAT_ConfigID se_Seed sh_Seed MA_Utility_MAE_uq_I9 -o
</hansl>
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
12:23 p.m.
On Wed, 20 Sep 2017, Schaff, Frederik wrote:
>> However, I can see that since you want to import hundreds of
series, "join"
>> is not going to be very convenient.
>
> Heh. I beg to differ :)
>
> Here's an alternative implementation which uses "join". Evaluating the
> relative merits of the two solutions is left as an exercise to the reader.
Ha, I suspected there could be a nice way of using "join" for the job,
and naturally Jack has found it!
Thanks. This is more concise (and in some points more general - the
rest I changed), but the join command does not work...? The problem
seems to be that
<hansl>
cmd = sprintf(" %s %s --ikey=ABMAT_ConfigID\n", fname, v)
</hansl>
Does somehow not parse "v" to cmd. If I print cmd, the content of
"v" is missing. If I only print v, it is fine. Is there a maximum
string length or such?
Ah, I think this is due to the invalid identifiers in your tsv files,
for example "Attendance_L-Skew_I1" (dashes/minus signs not allowed).
"join" is supposed to be able to handle this; the use of the "fixname"
function in Jack's script is the prescribed mechanism. But apparently
something is going wrong in this case. We'll investigate.
Allin
12:40 p.m.
>>> However, I can see that since you want to import
hundreds of series,
"join"
>>> is not going to be very convenient.
>>
>> Heh. I beg to differ :)
>>
>> Here's an alternative implementation which uses "join".
Evaluating
>> the relative merits of the two solutions is left as an exercise to the reader.
Ha, I suspected there could be a nice way of using "join" for the job, and
naturally Jack has found it!
> Thanks. This is more concise (and in some points more general - the
> rest I changed), but the join command does not work...? The problem
> seems to be that
>
> <hansl>
> cmd = sprintf(" %s %s --ikey=ABMAT_ConfigID\n", fname, v) </hansl>
>
> Does somehow not parse "v" to cmd. If I print cmd, the content of
"v"
> is missing. If I only print v, it is fine. Is there a maximum string
> length or such?
Ah, I think this is due to the invalid identifiers in your tsv files, for example
"Attendance_L-Skew_I1" (dashes/minus signs not allowed).
"join" is supposed to be able to handle this; the use of the
"fixname"
function in Jack's script is the prescribed mechanism. But apparently
something is going wrong in this case. We'll investigate.
Allin
Many thanks! In the meantime I'll avoid names with "-". PS: I learned a lot
from both scripts. :)
Best
Frederik
6:48 p.m.
On Wed, 20 Sep 2017, Schaff, Frederik wrote:
Many thanks! In the meantime I'll avoid names with "-".
PS: I
learned a lot from both scripts. :)
Glad to help. In fact I was wrong, the problem you found with Jack's
script was not to do with the identifiers, and after a further tweak
in git (and today's snapshots) it should work fine.
Also after recent tweaks your original idea of using "append"
will work -- on condition that you replace the label "ABMAT_ConfigID"
with "obs" (thus getting gretl to think of the values as observation
numbers). On *nix that could be done via
for f in *.tsv ; do
sed -i s/ABMAT_ConfigID/obs/ $f
done
Then the following (which borrows from Jack) will do the job:
<hansl>
set verbose off
string path_base = "" # or whatever
string path_ts = path_base ~ "parameters_"
string ts_name = ".tsv"
scalar first = 1
scalar last = 2 #1600
fname = sprintf("%s%d%s",path_ts,first,ts_name)
open @fname --preserve
loop i=first+1..last -q
fname = sprintf("%s%d%s",path_ts,i,ts_name)
append @fname --quiet
endloop
strings intervals =
defarray("0-499","500-999","1000-1499",\
"1500-1999","1500-1504","1500-1519","1500-1549",\
"1500-1599","1500-1749")
scalar n = nelem(intervals)
loop j=first..last --quiet
smpl j j
loop i=1..n --quiet
fname = sprintf("%sstat_%s_%d%s",path_base,intervals[i],j,ts_name)
append @fname --update-overlap
endloop
endloop
smpl full
varlist
print se_Seed sh_Seed MA_Utility_MAE_uq_I9 -o
</hansl>
On the relative merits of the different approaches: the matrix method
is fastest, but the join method is most "secure" in that it would
expose any inconsistencies in the input files. (All that the matrix
method checks is that the successive data files are of the right
overall dimensions to pack into place.)
Allin
8:04 a.m.
Hi there,
unfortunately this works only for 2 but not 3 or more records... And I do not understand
why.
I tried Jack's idea again (see attached, it's a bit long), the problem remains
that the long string "v" holding all the string names cannot be printed. I found
that 'printf "%s",v' does not work but 'print v' does... That is
the reason why the 'join @cmd' does not hold the variable names.
Best
Frederik
-----Ursprüngliche Nachricht-----
Von: gretl-users-bounces(a)lists.wfu.edu [mailto:gretl-users-
bounces(a)lists.wfu.edu] Im Auftrag von Allin Cottrell
Gesendet: Mittwoch, 20. September 2017 20:49
An: Gretl list <gretl-users(a)lists.wfu.edu>
Betreff: Re: [Gretl-users] WG: append horizontal and vertical
On Wed, 20 Sep 2017, Schaff, Frederik wrote:
> Many thanks! In the meantime I'll avoid names with "-". PS: I learned
> a lot from both scripts. :)
Glad to help. In fact I was wrong, the problem you found with Jack's script
was not to do with the identifiers, and after a further tweak in git (and
today's snapshots) it should work fine.
Also after recent tweaks your original idea of using "append"
will work -- on condition that you replace the label "ABMAT_ConfigID"
with "obs" (thus getting gretl to think of the values as observation numbers).
On *nix that could be done via
for f in *.tsv ; do
sed -i s/ABMAT_ConfigID/obs/ $f
done
Then the following (which borrows from Jack) will do the job:
<hansl>
set verbose off
string path_base = "" # or whatever
string path_ts = path_base ~ "parameters_"
string ts_name = ".tsv"
scalar first = 1
scalar last = 2 #1600
fname = sprintf("%s%d%s",path_ts,first,ts_name)
open @fname --preserve
loop i=first+1..last -q
fname = sprintf("%s%d%s",path_ts,i,ts_name)
append @fname --quiet
endloop
strings intervals =
defarray("0-499","500-999","1000-1499",\
"1500-1999","1500-1504","1500-1519","1500-1549",\
"1500-1599","1500-1749")
scalar n = nelem(intervals)
loop j=first..last --quiet
smpl j j
loop i=1..n --quiet
fname = sprintf("%sstat_%s_%d%s",path_base,intervals[i],j,ts_name)
append @fname --update-overlap
endloop
endloop
smpl full
varlist
print se_Seed sh_Seed MA_Utility_MAE_uq_I9 -o </hansl>
On the relative merits of the different approaches: the matrix method is
fastest, but the join method is most "secure" in that it would expose any
inconsistencies in the input files. (All that the matrix method checks is that
the successive data files are of the right overall dimensions to pack into
place.)
Allin
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
6:01 p.m.
On Thu, 21 Sep 2017, Schaff, Frederik wrote:
unfortunately this works only for 2 but not 3 or more records... And
I do not understand why.
If you can send me the third-step files I'll try testing here.
I tried Jack's idea again (see attached, it's a bit long),
the
problem remains that the long string "v" holding all the string
names cannot be printed. I found that 'printf "%s",v' does not work
but 'print v' does... That is the reason why the 'join @cmd' does
not hold the variable names.
Hmm, this is quite odd. There's no such problem on Linux but I tried
testing on Windows and I now see what you mean. There seems to be a
string-length problem with [s]printf on Windows; I'll investigate.
In the meantime there's a workaround: the script executes OK if you
piece together the command for "join" like this:
cmd = sprintf("%s ", fname)
cmd ~= v
cmd ~= sprintf("--ikey=%s --okey=%s", in_key, out_key)
Allin
7:44 p.m.
> unfortunately this works only for 2 but not 3 or more records...
And I
> do not understand why.
If you can send me the third-step files I'll try testing here.
You find the files attached.
> I tried Jack's idea again (see attached, it's a bit
long), the problem
> remains that the long string "v" holding all the string names cannot
> be printed. I found that 'printf "%s",v' does not work but
'print v'
> does... That is the reason why the 'join @cmd' does not hold the
> variable names.
Hmm, this is quite odd. There's no such problem on Linux but I tried testing
on Windows and I now see what you mean. There seems to be a string-
length problem with [s]printf on Windows; I'll investigate.
In the meantime there's a workaround: the script executes OK if you piece
together the command for "join" like this:
cmd = sprintf("%s ", fname)
cmd ~= v
cmd ~= sprintf("--ikey=%s --okey=%s", in_key, out_key)
Allin
Thanks a lot! This work-around does the job. And sorry for all the inconvenience...
Frederik
12:34 a.m.
On Thu, 21 Sep 2017, Schaff, Frederik wrote:
>> unfortunately this works only for 2 but not 3 or more
records... And I
>> do not understand why.
>
> If you can send me the third-step files I'll try testing here.
You find the files attached.
Thanks, I'll take a look as soon as I can.
>> I tried Jack's idea again (see attached, it's a bit
long), the
>> problem remains that the long string "v" holding all the string
>> names cannot be printed. I found that 'printf "%s",v' does not
>> work but 'print v' does... That is the reason why the 'join
>> @cmd' does not hold the variable names.
>
> Hmm, this is quite odd. There's no such problem on Linux but I
> tried testing on Windows and I now see what you mean. There seems
> to be a string- length problem with [s]printf on Windows; I'll
> investigate.
>
> In the meantime there's a workaround: the script executes OK if
> you piece together the command for "join" like this:
>
> cmd = sprintf("%s ", fname)
> cmd ~= v
> cmd ~= sprintf("--ikey=%s --okey=%s", in_key, out_key)
Thanks a lot! This work-around does the job. And sorry for all the
inconvenience...
No inconvenience, we want gretl's commands and functions to work as
advertised on all platforms, and so we thank you for pointing out
problems so we can fix them. Both "append" and "join" should come
out of this in better shape.
Meanwhile, I think I'm closing in on the problem with Jack's "join"
script on Windows (not Jack's fault) and we should have a proper fix
available soon.
Allin
6:10 p.m.
On Thu, 21 Sep 2017, Allin Cottrell wrote:
On Thu, 21 Sep 2017, Schaff, Frederik wrote:
> I tried Jack's idea again (see attached, it's a bit long), the problem
> remains that the long string "v" holding all the string names cannot be
> printed. I found that 'printf "%s",v' does not work but 'print
v' does...
> That is the reason why the 'join @cmd' does not hold the variable names.
Hmm, this is quite odd. There's no such problem on Linux but I tried testing
on Windows and I now see what you mean. There seems to be a string-length
problem with [s]printf on Windows; I'll investigate.
I'm glad you exposed this problem. Gretl's "printf" and
"sprintf" are
implemented internally using the C-library function vsnprintf and it
turns out we were missing something: on Windows vsnprintf doesn't
conform to the ISO C standard by default; you have to define a special
macro to get standard-conforming behavior. We're now doing that (in
git and snapshots). So the workaround I described is no longer needed.
Allin
8:17 p.m.
On Thu, 21 Sep 2017, Schaff, Frederik wrote:
unfortunately this works only for 2 but not 3 or more records...
And I do not understand why.
Frederik is referring to my script to do his data concatenation
job using the (recently tweaked) "append" command; see
http://lists.wfu.edu/pipermail/gretl-users/2017-September/012690.html
for details.
I see what the problem is. First, there was a residual issue with
the "append" command, which I have now fixed in git. But beyond
that, I was assuming a greater regularity in Frederik's data files
than actually obtains. The following comment will mean little
to anyone who hasn't tried working with Frederik's data, but here
goes...
My assumption was that the per-observation files
stat_0-499_1.tsv
stat_0-499_2.tsv
stat_0-499_3.tsv
(and other such tuples) contained observations (1,2,3) on a common
set of variables. Mostly that does seem to be the case -- and if it
were always the case my "append" recipe should work -- but it's not
always the case. In particular the file stat_0-499_3.tsv contains
three variable identifiers that are not found in any other file,
namely
Attendance_L_cv_I1
Attendance_L_Skew_I1
Attendance_L_Kurt_I1
This discrepancy means the "append" (a fairly simple-minded command)
will not do the job; the other options are "join", as Jack showed
in
http://lists.wfu.edu/pipermail/gretl-users/2017-September/012686.html
or handling the data as matrices, as I showed in
http://lists.wfu.edu/pipermail/gretl-users/2017-September/012683.html
(Jack's suggestion was subject to Windows bug which has now been
fixed.)
Allin
9:08 p.m.
On Fri, 22 Sep 2017, Allin Cottrell wrote:
On Thu, 21 Sep 2017, Schaff, Frederik wrote:
> unfortunately this works only for 2 but not 3 or more records... And I do
> not understand why.
Frederik is referring to my script to do his data concatenation job using the
(recently tweaked) "append" command; see
http://lists.wfu.edu/pipermail/gretl-users/2017-September/012690.html
for details.
I see what the problem is. First, there was a residual issue with the
"append" command, which I have now fixed in git. But beyond that, I was
assuming a greater regularity in Frederik's data files than actually obtains.
The following comment will mean little
to anyone who hasn't tried working with Frederik's data, but here goes...
My assumption was that the per-observation files
stat_0-499_1.tsv
stat_0-499_2.tsv
stat_0-499_3.tsv
(and other such tuples) contained observations (1,2,3) on a common set of
variables. Mostly that does seem to be the case -- and if it were always the
case my "append" recipe should work -- but it's not always the case. In
particular the file stat_0-499_3.tsv contains three variable identifiers that
are not found in any other file, namely
Attendance_L_cv_I1
Attendance_L_Skew_I1
Attendance_L_Kurt_I1
This discrepancy means the "append" (a fairly simple-minded command) will not
do the job [...]
Ah, but now I understand the source of the discrepancy, and it
should be fixable, if not already fixed, by Frederik. The data files
that he posted originally (the "1" and "2" files) contained the
invalid identifiers:
Attendance_L-cv_I1
Attendance_L-Skew_I1
Attendance_L-Kurt_I1
These are invalid because "-" is an operator (minus) from gretl's
point of view and you can't stick operators into identifiers. So
gretl "auto-corrected" these names to
Attendance_Lcv_I1
Attendance_LSkew_I1
Attendance_LKurt_I1
(just dropping the offending character, as explained in the doc for
gretl's fixname() function). Meanwhile, however, Frederik himself
corrected these names to
Attendance_L_cv_I1
Attendance_L_Skew_I1
Attendance_L_Kurt_I1
(replacing '-' with '_') in the "3" file that he sent more
recently.
So I presume that with current gretl git/snapshot my "append" recipe
should work after all, provided one is working with a consistent set
of Frederik's data files (either all auto-corrected in gretl's
manner, or all manually corrected in an alternative but consistent
manner).
Allin
9:48 p.m.
On Fri, 22 Sep 2017, Allin Cottrell wrote:
So I presume that with current gretl git/snapshot my
"append" recipe should
work after all, provided one is working with a consistent set of Frederik's
data files (either all auto-corrected in gretl's manner, or all manually
corrected in an alternative but consistent manner).
It should be noted that with the file/string functions we've been adding
over time it is entirely possible to pre-scan the header of a CSV file to
pre-parse the variable names if necessary.
... which brings me to a something I've been thinking about recently while
working in R: they have this nice head() function (and tail(), for that
matter) which can be handy sometimes. How would you like a --head option
to the "print" command? Or maybe, a little function for the extra package,
like
<hansl>
function void head(list X, int n[0::10])
outfile buf --buffer
print X --byobs
outfile --close
l = ""
i = -3
loop while getline(buf, l) && i<n --quiet
printf "%s\n", l
i++
endloop
end function
</hansl>
(Of course the above would be neater to handle via matrices, but then you
have NAs and strings to take care of...)
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
10:15 p.m.
So I presume that with current gretl git/snapshot my "append" recipe should
work after all, provided one is working with a consistent set of Frederik's data
files (either all auto-corrected in gretl's manner, or all manually corrected in
an alternative but consistent manner).
Dear Allin,
I finally checked it all with the current snapshot (todays). The join command via the
matrix now works fine without the workaround. The append is faster and - given that my
"identifier" is "obs" - works fine, too.
If in future one could provide a unique identifier as an option in the append command
(perhaps even inkey and outkey, as in join), that would be even more user friendly ;).
Many thanks!
Frederik
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
2667
days inactive
2675
days old
22 comments
4 participants
participants (4)
-
Allin Cottrell -
Riccardo (Jack) Lucchetti -
Schaff, Frederik -
Sven Schreiber