5:01 p.m.
Hi,
I stumbled across a very strange thing, namely ending up with two
variables with the same name "nordwest" in a panel workfile. I'm
attaching a stripped-down version in the hope that the effect is still
visible when the file gets to you.
The source was a Stata file, but I just re-imported to check, and only
one "nordwest" appears right after the import. So it must have been
something after that, within gretl. BTW, the content of the variables
were not identical, which is of course even worse.
This is with the 32bit gretl 1.9.14 on Win7.
Thanks,
sven
5:33 p.m.
Am 31.12.2013 18:01, schrieb Sven Schreiber:
Hi,
I stumbled across a very strange thing, namely ending up with two
variables with the same name "nordwest" in a panel workfile. I'm
attaching a stripped-down version in the hope that the effect is still
visible when the file gets to you.
The source was a Stata file, but I just re-imported to check, and only
one "nordwest" appears right after the import. So it must have been
something after that, within gretl. BTW, the content of the variables
were not identical, which is of course even worse.
This is with the 32bit gretl 1.9.14 on Win7.
Thanks,
sven
Further information on this: in the original imported file, the variable
with the next ID number after "nordwest" was called "südwest" (with
u-umlaut) which was obviously not imported correctly, there was some
strange sign instead of the umlaut. Possibly this got interpreted by
gretl as something weird, which caused the mess-up.
cheers,
sven
1:57 a.m.
On Tue, 31 Dec 2013, Sven Schreiber wrote:
Further information on this: in the original imported file, the
variable
with the next ID number after "nordwest" was called "südwest" (with
u-umlaut) which was obviously not imported correctly, there was some
strange sign instead of the umlaut. Possibly this got interpreted by
gretl as something weird, which caused the mess-up.
Text encoding in Stata files is quite primitive. Stata doesn't
support UTF-8, and apparently encodes non-ASCII characters in the
locale MS-DOS "code page" -- without, so far as I can tell,
recording the specific encoding used in the .dta file.
That said, I've now modified the Stata importer in CVS to use the
libgretl function iso_to_ascii() on Stata variable names. Hopefully,
in most cases this should manage to convert "südwest" to "sudwest"
without knowledge of the actual encoding used.
(Googling this issue reveals some frustration among Stata users in
non-English locales. It also reveals that a prominent Stata guru
apparently believes that UTF-8 performs the magic of representing
all Unicode characters in an 8-bit encoding, a misapprehension which
does not bode well for Stata's eventually supporting UTF-8.)
Allin
12:55 p.m.
Am 01.01.2014 02:57, schrieb Allin Cottrell:
On Tue, 31 Dec 2013, Sven Schreiber wrote:
> Further information on this: in the original imported file, the variable
> with the next ID number after "nordwest" was called "südwest"
(with
> u-umlaut) which was obviously not imported correctly, there was some
> strange sign instead of the umlaut. Possibly this got interpreted by
> gretl as something weird, which caused the mess-up.
Text encoding in Stata files is quite primitive. Stata doesn't support
UTF-8, and apparently encodes non-ASCII characters in the locale MS-DOS
"code page" -- without, so far as I can tell, recording the specific
encoding used in the .dta file.
That said, I've now modified the Stata importer in CVS to use the
libgretl function iso_to_ascii() on Stata variable names. Hopefully, in
most cases this should manage to convert "südwest" to "sudwest"
without
knowledge of the actual encoding used.
Thanks, this should help in most cases. According to the table on
http://en.wikipedia.org/wiki/Windows-1252 I guess that the remaining
problematic letters would be "contracted oe" (unicode 0152 and 0153),
and s and z with "inverted hats" (unicode 0160, 0161, 017D, 017E). Don't
know if these are valid characters in the respective local Stata versions.
Another primitive-but-all-purpose solution might be to replace any
non-Ascii characters with placeholders like "__"?
cheers,
sven
5:34 p.m.
On Tue, 31 Dec 2013, Sven Schreiber wrote:
Hi,
I stumbled across a very strange thing, namely ending up with two variables
with the same name "nordwest" in a panel workfile. I'm attaching a
stripped-down version in the hope that the effect is still visible when the
file gets to you.
The source was a Stata file, but I just re-imported to check, and only one
"nordwest" appears right after the import. So it must have been something
after that, within gretl. BTW, the content of the variables were not
identical, which is of course even worse.
This is with the 32bit gretl 1.9.14 on Win7.
This is in fact a bit disturbing, but is a consequence of the fact that
internally the two object are distinct. The fact that series have a name
is (surprising as it may sound) little more than a convenience for the
user. Internally, series are identified by their progressive order. If you
could send us the original Stata file, we could see why we end up with
duplicate labels and how we could modify the auto-renaming algorithm to
avoid similar cases in the future.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
1:03 p.m.
Am 31.12.2013 18:34, schrieb Riccardo (Jack) Lucchetti:
This is in fact a bit disturbing, but is a consequence of the fact
that
internally the two object are distinct. The fact that series have a name
is (surprising as it may sound) little more than a convenience for the
user. Internally, series are identified by their progressive order. If
you could send us the original Stata file, we could see why we end up
with duplicate labels and how we could modify the auto-renaming
algorithm to avoid similar cases in the future.
See the other messages, the problem is mostly solved. However, may I
insist that gretl "identifiers" (official parlance) be actually
identifying, and not just a convenience handle. Especially since the
internal ID numbers are not time-invariant, either.
I was going to suggest an additional checking layer that ensures
uniqueness of identifiers, but OTOH it's not quite clear where that
layer should reside I guess, and this problem that I reported also seems
very special. And before I have never encountered a non-uniqueness
problem like this.
BTW, happy new year!
Sven
3:35 a.m.
On Wed, 1 Jan 2014, Sven Schreiber wrote:
Am 31.12.2013 18:34, schrieb Riccardo (Jack) Lucchetti:
> This is in fact a bit disturbing, but is a consequence of the fact that
> internally the two object are distinct. The fact that series have a name
> is (surprising as it may sound) little more than a convenience for the
> user. Internally, series are identified by their progressive order. If
> you could send us the original Stata file, we could see why we end up
> with duplicate labels and how we could modify the auto-renaming
> algorithm to avoid similar cases in the future.
See the other messages, the problem is mostly solved. However, may I
insist that gretl "identifiers" (official parlance) be actually
identifying, and not just a convenience handle. Especially since the
internal ID numbers are not time-invariant, either.
Names of series _should_ indeed be uniquely identifying; that is
certainly the intent. Neither the names nor the ID numbers of series
are immutable: you can rename a series, and a series of a given name
can be renumbered if a lower-numbered series is deleted.
Nonetheless, names ought to be unique (across all gretl objects, not
just series) and there's code in place to try to ensure that.
Apparently the code was defeated in this instance -- I'm not sure
how, and I haven't heard of it happening before. (Well, to be
precise, I have seen it before on rare occasions in the relatively
distant past, in consequence of identifiable bugs that were
subsequently fixed.)
I'd second Jack's suggestion: if you can send us the original .dta
file maybe we can find out how the duplicate name came into being,
though we might also need a hint as to what sort of commands were
executed after opening the Stata file.
Allin
6:57 p.m.
Am 02.01.2014 04:35, schrieb Allin Cottrell:
On Wed, 1 Jan 2014, Sven Schreiber wrote:
I'd second Jack's suggestion: if you can send us the original .dta
file maybe we can find out how the duplicate name came into being,
though we might also need a hint as to what sort of commands were
executed after opening the Stata file.
The file is too big and also restricted.
But I just found out that it's enough to do:
1) import
2) save as gretl file (undated)
-- the imported but corrupted variable name is still visible in the
variable list--
3) close the file
4) open the file again, and then I have a double "nordwest"!
What's also interesting, if right after the import (step 1 above) I give
the command 'varlist', I get (excerpt):
...
1149) rente1 1150) rente2 1151) rente3
1152) netto 1153) netto1 1154) netto2
1158) nation 1159) deutsch 1160) deutsch_1
...
Notice that the row with IDs 1155-1157 is missing -- 1156 is "nordwest",
and 1157 is the problematic ex-"südwest".
Further symptoms right after the import/ before saving:
If I open the series to show the values, in that new window gretl shows
the correctly spelled "südwest" header, in contrast to the mess-up in
the variable list in the main window.
If I then copy that header string into the Scite text editor and change
the encoding to UTF-8, I get a mark 'xFC' instead of the 'ü' (u-umlaut)
in the middle of the string. Of course xFC /252 is the u-umlaut in the
Windows codepage, but in UTF-8 it's an invalid byte AFAI understand.
If instead of opening the series/values window I open the attributes
window, the messed-up name appears there. There I can only select the
first two characters (not the trailing "dwest"), and if I try to copy
that (ctrl-c), gretl.exe crashes!
Hope this is enough information,
sven
8:26 p.m.
On Thu, 2 Jan 2014, Sven Schreiber wrote:
Am 02.01.2014 04:35, schrieb Allin Cottrell:
> On Wed, 1 Jan 2014, Sven Schreiber wrote:
>
>
> I'd second Jack's suggestion: if you can send us the original .dta
> file maybe we can find out how the duplicate name came into being,
> though we might also need a hint as to what sort of commands were
> executed after opening the Stata file.
>
The file is too big and also restricted.
OK.
But I just found out that it's enough to do:
1) import
2) save as gretl file (undated)
-- the imported but corrupted variable name is still visible in the
variable list--
3) close the file
4) open the file again, and then I have a double "nordwest"!
Thanks. I wonder if this is fixed by the revision to stata_import.c?
That's in CVS but not yet in the Windows snapshots. (I hope to get a
new snapshot in place tomorrow or the day after.)
This is quite interesting, from Sergiy Radyakin
http://www.stata.com/statalist/archive/2013-10/msg00289.html :
"the variable names in Stata must be comprised of letters a..z,
A..Z, digits 0..9 and an underscore, and may not start with a digit.
Use of any other symbols is NOT supported. Some conversion programs
(including one of my own) used to produce files with accents/umlauts
and other non ASCII latin characters if they were present in the
original file. Although Stata can open such a file, some commands
will not work later. Non-Stata software that actually validates the
input may refuse to work with such files. If anything, it is a bug
and Stata should properly respond with error 198 .. invalid name."
Allin
10:04 p.m.
On Tue, 31 Dec 2013, Sven Schreiber wrote:
I stumbled across a very strange thing, namely ending up with two
variables
with the same name "nordwest" in a panel workfile. I'm attaching a
stripped-down version in the hope that the effect is still visible when the
file gets to you.
Yes, the effect is visible, but I'm afraid it offers no clue as to
how it happened.
The source was a Stata file, but I just re-imported to check, and
only one
"nordwest" appears right after the import. So it must have been something
after that, within gretl.
You mean some command that you executed, that resulted in the
creation of another variable of the same name?
Allin
4036
days inactive
4038
days old
9 comments
3 participants
participants (3)
-
Allin Cottrell -
Riccardo (Jack) Lucchetti -
Sven Schreiber