8:34 a.m.
This has been dabeted several times, but IMO the issue still remains.
A colleague was trying to use gretl to import a dataset (with extension
.txt, but that doesn't really matter) whose first lines look like this:
"num"|"x"|"y"|"z"
"1"|0.273856207444503|-0.036672201779096|0.854726937504524
"2"|2.24481689268639|-0.546030016142993|0.23406506726493
"3"|-0.125423034417563|-0.606878201926287|-0.853921693817499
"4"|-0.54357344128812|1.10835827274681|-0.104247994046094
"5"|-1.45939839398061|-0.274494522530397|0.111905961611328
Eventually, we managed to solve the problem by manually editing the file
and replacing pipe characters by commas, but it would have been nice to
have an option to give the CSV importer the delimiter character to split
columns.
This would also solve cases when you get a mish-mash of commas and
semi-colons, like in (pseudo-)CSV files produced by national versions of
some spreadsheets.
Allin, I think I remember you were against this feature for some reason,
but I can't recall what it was. Could you please remind me? Thanks!
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
4:59 p.m.
On Thu, Nov 18, 2021 at 8:34 AM Riccardo (Jack) Lucchetti
<p002264(a)staff.univpm.it> wrote:
This has been dabeted several times, but IMO the issue still remains.
A colleague was trying to use gretl to import a dataset (with extension
.txt, but that doesn't really matter) whose first lines look like this:
"num"|"x"|"y"|"z"
"1"|0.273856207444503|-0.036672201779096|0.854726937504524
[ the "pipe" symbol '|' is being used as the field separator ]
it would have been nice to have an option to give the CSV importer the
delimiter character to split columns.
Allin, I think I remember you were against this feature for some reason,
but I can't recall what it was. Could you please remind me? Thanks!
Well, my reluctance was based on the supposition that in maybe 99+% of cases
libgretl would know what separator to use better than most users. I've now added
'|' as one of the options libgretl should consider, and in my testing
pipe-separated
fields work fine. I'm willing to reconsider, but I'd be surprised if
anyone can come
up with a real-world case where gretl (now) gets it wrong.
Allin
3:25 a.m.
On Fri, 19 Nov 2021, Cottrell, Allin wrote:
Well, my reluctance was based on the supposition that in maybe 99+%
of
cases libgretl would know what separator to use better than most users.
I've now added '|' as one of the options libgretl should consider, and
in my testing pipe-separated fields work fine. I'm willing to
reconsider, but I'd be surprised if anyone can come up with a real-world
case where gretl (now) gets it wrong.
And I would too. All I'm saying is, why not give the user the option to
use the automatic algorithm (as default) or force the delimiter choice for
the remaining 1% of the cases?
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
9:13 p.m.
On Sat, 20 Nov 2021, Riccardo (Jack) Lucchetti wrote:
On Fri, 19 Nov 2021, Cottrell, Allin wrote:
> Well, my reluctance was based on the supposition that in maybe 99+% of
> cases libgretl would know what separator to use better than most users.
> I've now added '|' as one of the options libgretl should consider, and in
> my testing pipe-separated fields work fine. I'm willing to reconsider, but
> I'd be surprised if anyone can come up with a real-world case where gretl
> (now) gets it wrong.
And I would too. All I'm saying is, why not give the user the option to use
the automatic algorithm (as default) or force the delimiter choice for the
remaining 1% of the cases?
Feel free to create a branch and implement that. I won't stand in
the way if the team reckons it's worthwhile.
Allin
6:17 a.m.
On Mon, 22 Nov 2021, Allin Cottrell wrote:
On Sat, 20 Nov 2021, Riccardo (Jack) Lucchetti wrote:
> On Fri, 19 Nov 2021, Cottrell, Allin wrote:
>
>> Well, my reluctance was based on the supposition that in maybe 99+% of
>> cases libgretl would know what separator to use better than most users.
>> I've now added '|' as one of the options libgretl should consider,
and in
>> my testing pipe-separated fields work fine. I'm willing to reconsider, but
>> I'd be surprised if anyone can come up with a real-world case where gretl
>> (now) gets it wrong.
>
> And I would too. All I'm saying is, why not give the user the option to use
> the automatic algorithm (as default) or force the delimiter choice for the
> remaining 1% of the cases?
Feel free to create a branch and implement that. I won't stand in the way if
the team reckons it's worthwhile.
Update: after inspecting the code, I found that what I had set out to
implement is already there, except with a different twist. The "csv_delim"
setting also affects csv reading: first, the currently set delimiter is
tried, and we revert to the automatic heuristic algorithm on failure. This
is not 100% complete, since it leaves out cases when the column separator
is not one of the traditional ones (comma, space, etc), but I can
personally live with that quite happily (I have no plan to save LaTeX
tabulars as CSV files).
Instead, I ended up making a very slight change (now in git) that makes
gretl a little more tolerant with malformed CSV files. Until now, we
aborted CSV import if we found a line where the number of fields was
different from the one we had previously calculated. After the change, we
only abort if we hit a line with MORE entries than the number of columns.
When the number of fields on a given CSV line is FEWER than necessary, we
now print out a warning and fill the missing entries with NAs.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
1031
days inactive
1036
days old
4 comments
3 participants
participants (3)
-
Allin Cottrell -
Cottrell, Allin
-
Riccardo (Jack) Lucchetti