2:09 p.m.
User Guide 4.4 refers to the `import` command.
It does not appear in the command ref.
If you go to the console and type::
help import
you are told `import` is not a gretl command.
It is of course, but apparently undocumented?
I'd like to import data from a CSV file.
I'd like to do this in a script.
In addition to being able to specify the date
range,I need the following functionality:
skiplines: the number of lines to skip at the
beginning of the file
usecols: which columns to use for data
varnames: specify names (which are not in the CSV file)
Possible?
Finally, if specifying the date range is not possible,
is it possible to reset the dataset structure at the
command line (rather than via menus)?
Thanks,
Alan
2:31 p.m.
2009/9/23 Alan G Isaac <alan.isaac(a)gmail.com>
I'd like to import data from a CSV file.
I'd like to do this in a script.
In addition to being able to specify the date
range,I need the following functionality:
skiplines: the number of lines to skip at the
beginning of the file
usecols: which columns to use for data
varnames: specify names (which are not in the CSV file)
Possible?
Finally, if specifying the date range is not possible,
is it possible to reset the dataset structure at the
command line (rather than via menus)?
I'm not sure about CSV files, but Gretl can open spreadsheet files
(Gnumeric, Open Document and Excel) and plain text in the way you want. Look
at page 39 in the Gretl Command Reference. Example:
open "C:\yourdata.xls" --sheet=3 --rowoffset=1 coloffset=1
Tell me if it doesn't work, ok?
Best regards,
Henrique C. de Andrade
Doutorando em Economia Aplicada
Universidade Federal do Rio Grande do Sul
www.ufrgs.br/ppge
3:22 p.m.
On 9/23/2009 2:31 PM, Henrique wrote:
I'm not sure about CSV files, but Gretl can open spreadsheet
files
(Gnumeric, Open Document and Excel) and plain text in the way you
want. Look at page 39 in the Gretl Command Reference. Example:
open "C:\yourdata.xls" --sheet=3 --rowoffset=1 coloffset=1
Those options appear to be ignored for CSV files, unfortunately.
But thanks.
Alan
2:47 p.m.
On Wed, 23 Sep 2009, Alan G Isaac wrote:
User Guide 4.4 refers to the `import` command.
It does not appear in the command ref.
If you go to the console and type::
help import
you are told `import` is not a gretl command.
It is of course, but apparently undocumented?
Sorry, the Guide needs to be updated: "import" was once a gretl
command but it was subsumed under "open" some time ago. The reason
"import" will still work as a command is that it is declared
(internally) as an alias for "open" in order to preserve backward
compatibility. We need to replace all references to an "import"
command in the Guide.
I'd like to import data from a CSV file.
I'd like to do this in a script.
In addition to being able to specify the date
range,I need the following functionality:
skiplines: the number of lines to skip at the
beginning of the file
usecols: which columns to use for data
varnames: specify names (which are not in the CSV file)
Possible?
Well, the --cols option to "open" lets you set the specs for
reading columns, and may do what you want. We don't have a
"skiplines" option, though lines should automatically be skipped
if they start with '#'. One could fairly easily write a shell
command to preprocess the data and execute it via "! <shell>"
in gretl.
If no varnames are present the series will be named v1, v2, etc.,
and you can use "rename" to attach meaningful names.
Finally, if specifying the date range is not possible,
is it possible to reset the dataset structure at the
command line (rather than via menus)?
Yes, via setobs (as you noted).
Allin.
3:33 p.m.
On 9/23/2009 2:47 PM, Allin Cottrell wrote:
One could fairly easily write a shell
command to preprocess the data and execute it via "!<shell>"
in gretl.
Sure, but I'm trying to help students get IFS data into gretl.
They have been generally encouraged to use CSV files,
so I'm trying to give them gretl-based instructions for getting
IFS produced CSV files into gretl.
If no varnames are present the series will be named v1, v2, etc.,
and you can use "rename" to attach meaningful names.
Yes, that is a fine option.
But skiprows (or a working rowoffset for CSV) and usecols would be very
helpful.
Fwiw.
Cheers,
Alan
8:21 p.m.
On Wed, 23 Sep 2009, Alan G Isaac wrote:
On 9/23/2009 2:47 PM, Allin Cottrell wrote:
> One could fairly easily write a shell
> command to preprocess the data and execute it via "!<shell>"
> in gretl.
Sure, but I'm trying to help students get IFS data into gretl.
They have been generally encouraged to use CSV files, so I'm
trying to give them gretl-based instructions for getting IFS
produced CSV files into gretl.
Gretl-based instructions are not necessarily exclusive of shell
commands, since (a) gretl can do shell, and (b) the shell is
capable of more flexible manipulation of text files than gretl can
ever be (more so on *nix than Windows, of course, but even on
Windows).
For example, if you want to discard the first 6 lines of a data
file, data.csv, on Windows, in gretl script/console:
! more +6 data.csv > gretldata.csv
open gretldata.csv
There's little point in trying to have gretl reinvent shell
processing. (With spreadsheet files, which are not plain text and
hence cannot easily be manipulated via the shell, we try to be
accommodating.)
Allin Cottrell
9:22 p.m.
On 9/23/2009 8:21 PM, Allin Cottrell wrote:
Gretl-based instructions are not necessarily exclusive of shell
commands, since (a) gretl can do shell, and (b) the shell is
capable of more flexible manipulation of text files than gretl can
ever be (more so on *nix than Windows, of course, but even on
Windows).
Hi Allin,
So I am unsure how much you want to hear about this, but I will respond
once.
It is really infeasible to introduce my undergrads to shell commands.
They are already shell shocked by my insistence that they write gretl
scripts.
Most of them have no idea what a shell is. And the shell is not cross
platform.
The cross-platform character of gretl is crucial to my choosing it.
I would have to communicate to them that "Mac users do it this way,
and Win users do it that way". This is just not going to go down well.
That's the reality I face.
There is also a different kind of argument. Sure CSV files are plain
text and
can therefore be easily manipulated by a variety of tools. But they are
also
a core spreadsheet format. Thus the --rowoffset and --coloffset options
of `open` should work with CSV too. Or so I claim, despite offering no
code to do this.
I'll shut up now.
Alan
PS It might interest you to know that my current class is split about
half/half
Mac/Windows, and that a fair number of the Windows users are using OO.
Just fyi. This is quite different than past years.
4:42 a.m.
Alan G Isaac schrieb:
There is also a different kind of argument. Sure CSV files are plain
text and
can therefore be easily manipulated by a variety of tools. But they are
also
a core spreadsheet format. Thus the --rowoffset and --coloffset options
of `open` should work with CSV too. Or so I claim, despite offering no
code to do this.
Hi Alan,
in principle I agree that this one's a valid point. However:
* --coloffset doesn't convince me in the case of csv files (as opposed
to --rowoffset in csv files, or --coloffset in spreadsheet files)
* my teaching experience is that it's even more manageable for students
if you tell them: ok, first open the file with a text editor (jedit is
cross-platform for example, but there are hundreds others of course),
insert # signs in the first couple of rows, then you're ready to go.
Many will like this (arguably inefficient) approach even more than
specifying --rowoffset in a script file.
BTW, I can tell you the data handling gets much worse when you live in a
locale with commas as standard decimal separators on the machines that
students and other software use. Then you need even more tricks,
especially with spreadsheet files. So while I think your request is
perfectly legitimate, you guys don't really have a problem ;-)
cheers,
sven
10:49 a.m.
On 9/24/2009 4:42 AM, Sven Schreiber wrote:
* --coloffset doesn't convince me in the case of csv files (as
opposed
to --rowoffset in csv files, or --coloffset in spreadsheet files)
Why? It is important to know that important data sources (like IFS)
use the first column for a date in non-standard format. The --coloffset
would allow ignoring this column.
I also insist that a CSV file *is* a spreadsheet file,
so the options valid for spreadsheet files should apply.
Alan Isaac
PS What I'd really like to do is:
--usecols=2:5,7,9:11,15:
10:53 a.m.
On Thu, 24 Sep 2009, Alan G Isaac wrote:
On 9/24/2009 4:42 AM, Sven Schreiber wrote:
> * --coloffset doesn't convince me in the case of csv files (as opposed
> to --rowoffset in csv files, or --coloffset in spreadsheet files)
Why? It is important to know that important data sources (like IFS)
use the first column for a date in non-standard format...
Could you send me a sample IFS CSV file so I can see what we're
dealing with? Thanks!
Allin.
11:08 a.m.
On 9/24/2009 10:53 AM, Allin Cottrell wrote:
Could you send me a sample IFS CSV file so I can see what we're
dealing with? Thanks!
Things to notice:
- date format (horrible! but easily regex'd)
- empty final column (typical output; don't know why)
- handling of missing values (see the end)
Cheers,
Alan
8:22 p.m.
Alan G Isaac schrieb:
On 9/24/2009 4:42 AM, Sven Schreiber wrote:
> * --coloffset doesn't convince me in the case of csv files (as opposed
> to --rowoffset in csv files, or --coloffset in spreadsheet files)
Why? It is important to know that important data sources (like IFS)
use the first column for a date in non-standard format. The --coloffset
would allow ignoring this column.
I also insist that a CSV file *is* a spreadsheet file,
so the options valid for spreadsheet files should apply.
hm yes, that makes sense I guess. So maybe you should add the
--coloffset (or --cols=...) thing to the feature request tracker on
gretl's sourceforge site, so that it doesn't get forgotten.
BTW, you said you don't want to use shell commands (for teaching)
because they're not cross-platform. Since I think you're a Pythonista
you could call Python from gretl as a kind of cross-platform shell
instead. Linux and Mac OS X come with python installed already, but of
course most of your students on windows would probably have to install
python first. Don't know if that's so bad, though.
The gretl command calling python via the shell would work something like
this:
! python -c
'f=open("myfile.csv");l=f.readlines()[offset:];f.close();o=open("file4gretl.csv","w");o.writelines(l);o.close()'
(all in one nasty long line -- don't know if the .close() things are
actually necessary here, and there are probably some bugs in there; and
don't forget to let your students enable shell commands in gretl's
preferences)
But this thing is for the rowoffset only of course. The coloffset would
require an even longer line of python commands I guess, possibly using
python's csv module.
good luck,
sven
11:35 p.m.
On Thu, 24 Sep 2009, Sven Schreiber wrote:
Alan G Isaac schrieb:
>
> There is also a different kind of argument. Sure CSV files
> are plain text and can therefore be easily manipulated by a
> variety of tools. But they are also a core spreadsheet
> format. Thus the --rowoffset and --coloffset options of
> `open` should work with CSV too. Or so I claim, despite
> offering no code to do this.
Hi Alan,
in principle I agree that this one's a valid point. However:
* --coloffset doesn't convince me in the case of csv files (as opposed
to --rowoffset in csv files, or --coloffset in spreadsheet files)
I'm willing to go with --coloffset in the case of CSV; the columns
are supposed to be well defined. But it may take a little while
to implement.
* my teaching experience is that it's even more manageable for
students if you tell them: ok, first open the file with a text
editor (jedit is cross-platform for example, but there are
hundreds others of course), insert # signs in the first couple
of rows, then you're ready to go. Many will like this (arguably
inefficient) approach even more than specifying --rowoffset in a
script file.
Now here I'm very much with Sven. And in fact I'm not so sure
that the suggested approach (checking and revising the CSV file in
a text editor before loading it into gretl) is inefficient; it may
well be optimal. This is a serious point.
I base this judgment on extensive experience with CSV files
generated by various statistical agencies around the world. My
experience tells me (a) that such files are often "broken" and
moreover (b) that they are not infrequently broken in ways that
you wouldn't expect. More than once I've thought that I had
figured out how to fix Agency X's broken CSV using a fairly simple
algorithm, only to find that _some_ of the files produced by X
were broken is a different way!
Therefore, I would not trust any statistical analysis based on
such data unless the researcher had actually opened the raw files
in a text editor and scanned them for problems.
DO NOT rely on simple algorithmic fixes, and don't suggest that
your students do so. Open the data and take a look. Comment out
lines you don't want. Use Find/Replace to get rid of idiocy such
as the ghost column
,""
appended to every line (well, maybe every line, but can you be
sure?) of IFS files.
Allin.
9:49 a.m.
On 9/24/2009 11:35 PM, Allin Cottrell wrote:
And in fact I'm not so sure
that the suggested approach (checking and revising the CSV file in
a text editor before loading it into gretl) is inefficient; it may
well be optimal. This is a serious point.
I take it seriously. I always look at my CSV files.
Cleaning them up with Vim is generally trivial.
But there are conflicting goals here. Do you realize that most undergrads
these days do not even know what a text editor is? I may not mind
teaching them that additional skill later in the semester, but I start
with a situation
where a few students freak out that they have to install gretl, and a fair
number cannot believe that they are asked to write a short script rather
than
to point and click their way to nirvana.
Alan
10:12 a.m.
On Fri, 25 Sep 2009, Alan G Isaac wrote:
On 9/24/2009 11:35 PM, Allin Cottrell wrote:
> And in fact I'm not so sure
> that the suggested approach (checking and revising the CSV file in
> a text editor before loading it into gretl) is inefficient; it may
> well be optimal. This is a serious point.
>
I take it seriously. I always look at my CSV files.
Cleaning them up with Vim is generally trivial.
But there are conflicting goals here. Do you realize that most
undergrads these days do not even know what a text editor is?
Yes, indeed I do :-(
I know you can't teach students everything at once. My approach
is to give them cleaned-up data at first, when they're just trying
to get familiar with econometric ideas, then later to reveal some
of the issues involved in the clean-up.
Allin
5349
days inactive
5352
days old
15 comments
4 participants
participants (4)
-
Alan G Isaac -
Allin Cottrell -
Henrique -
Sven Schreiber