10:37 a.m.
Hi,
I'm having problems on Windows using a file name with non-Latin
characters inside a foreign language R block, in the gretl.loadmat
function. If I stick to ASCII characters, it works. On Linux, it also works.
This is with a gretl snapshot and with R 3.5.1 (tried with 3.3.2
before), on Win 7. Don't know if there's anything gretl can do there?
thanks,
sven
1:23 p.m.
On Mon, 27 Aug 2018, Sven Schreiber wrote:
I'm having problems on Windows using a file name with non-Latin
characters
inside a foreign language R block, in the gretl.loadmat function. If I stick
to ASCII characters, it works. On Linux, it also works.
This is with a gretl snapshot and with R 3.5.1 (tried with 3.3.2 before), on
Win 7. Don't know if there's anything gretl can do there?
In the gretl script window all text is UTF-8, so the filename will be
passed to R in UTF-8 -- and therefore it will not work "as is" to
access the file on Windows, which doesn't do UTF-8.
I think the solution is (a) make sure your locale is set correctly
within R, using Sys.setlocale() if need be, and (b) tell R that the
incoming filename is in UTF-8. For the read.table() function this
would be a matter of appending the argument 'encoding="utf-8"',
as in
m <- as.matrix(read.table(fname, skip=1, encoding-"utf-8"))
If you can confirm that this works (for some UTF-8 path of your
choosing) we can add the encoding spec to the gretl.loadmat function
for R on Windows.
See also
https://stackoverflow.com/questions/5031630/how-to-source-r-file-saved-us...
Allin
1:35 p.m.
Am 27.08.2018 um 15:23 schrieb Allin Cottrell:
I think the solution is (a) make sure your locale is set correctly
within R, using Sys.setlocale() if need be, and (b) tell R that the
incoming filename is in UTF-8. For the read.table() function this
would be a matter of appending the argument 'encoding="utf-8"',
as in
m <- as.matrix(read.table(fname, skip=1, encoding-"utf-8"))
If you can confirm that this works (for some UTF-8 path of your
choosing) we can add the encoding spec to the gretl.loadmat function
for R on Windows.
OK, I will try that and report back.
BTW, in the guide in the section "Passing matrices from gretl to R" I
think it would be better to put the low-level read.table function in a
footnote and mention the convenience function gretl.loadmat first, as
the recommended way to do things. I can apply that change later myself,
though, if there are no objections.
thanks,
sven
2:36 p.m.
Am 27.08.2018 um 15:35 schrieb Sven Schreiber:
Am 27.08.2018 um 15:23 schrieb Allin Cottrell:
>
> I think the solution is (a) make sure your locale is set correctly
> within R, using Sys.setlocale() if need be, and (b) tell R that the
> incoming filename is in UTF-8. For the read.table() function this
> would be a matter of appending the argument 'encoding="utf-8"',
Hm, I'm having several doubts here: R's locale here is already set at
German_Germany.1252, which is correct AFAICT. But if I understand
correctly the filename on Windows on an NTFS disk is saved in UTF16? Not
sure how that is passed on to R by the OS.
> m <- as.matrix(read.table(fname, skip=1,
encoding-"utf-8"))
Next I believe the 'encoding' option in
read.table refers to the
contents of the file, not its name.
I have experimented with several things in R (iconv(), Encoding()), but
I'm not an R expert to know a lot about its non-Latin string handling,
so it hasn't worked. I guess I will go back to my old rule to only use
ASCII in file names... (In fact, I already have...)
thanks,
sven
2:49 p.m.
On Mon, 27 Aug 2018, Sven Schreiber wrote:
Am 27.08.2018 um 15:35 schrieb Sven Schreiber:
> Am 27.08.2018 um 15:23 schrieb Allin Cottrell:
>>
>> I think the solution is (a) make sure your locale is set correctly within
>> R, using Sys.setlocale() if need be, and (b) tell R that the incoming
>> filename is in UTF-8. For the read.table() function this would be a matter
>> of appending the argument 'encoding="utf-8"',
Hm, I'm having several doubts here: R's locale here is already set at
German_Germany.1252, which is correct AFAICT. But if I understand correctly
the filename on Windows on an NTFS disk is saved in UTF16? Not sure how that
is passed on to R by the OS.
The file should be accessible if its name is given either in the
locale encoding or UTF-16 (unless it's a "mixed language" name in
which case it would have to be in UTF-16).
>> m <- as.matrix(read.table(fname, skip=1,
encoding-"utf-8"))
Next I believe the 'encoding' option in read.table refers to the contents of
the file, not its name.
Ah, I suppose you're right. Then I guess the right thing would be to
recode the filename itself:
fname <- iconv(fname, from="utf-8", to="")
(where "" means the locale).
I haven't tested that yet but I'll explore a bit.
Allin
3:20 p.m.
Am 27.08.2018 um 16:49 schrieb Allin Cottrell:
fname <- iconv(fname, from="utf-8", to="")
I'm getting
"NA" as the result because the non-latin isn't possible to
convert to my locale.
I haven't tested that yet but I'll explore a bit.
Here's what I've tried unsuccessfully (whereas with Python there's no
problem):
-- But really, it's not a big deal, we can just avoid complicated file
names (or stick to Python)...
-sven
<hansl>
# test filenames and foreign
matrix m = {1,2}
mwrite(m, "замок.mat", 1)
### python
foreign language=python
fn = "замок.mat"
mp = gretl_loadmat(fn) # works!
print(mp)
end foreign
### R
foreign language=R
print(Sys.getlocale())
# gives: "LC_COLLATE=German_Germany.1252;LC_CTYPE=German_Germany.1252
... etc.
# (trying to set the locale to some UTF-8 thing didn't work)
fn <- "замок.mat"
print(fn)
fn <- paste(gretl.dotdir, fn, sep="")
print(fn)
fn <- iconv(fn, from="", to="UTF-8")
# (UTF-16 gives error, but this doesn't seem to help)
print(fn)
# mR <- as.matrix(read.table(fn, skip=1)) # fails
# mR <- gretl.loadmat(fn) # fails
end foreign
</hansl>
7:42 p.m.
On Mon, 27 Aug 2018, Sven Schreiber wrote:
Am 27.08.2018 um 16:49 schrieb Allin Cottrell:
>
> fname <- iconv(fname, from="utf-8", to="")
I'm getting "NA" as the result because the non-latin isn't possible to
convert to my locale.
> I haven't tested that yet but I'll explore a bit.
Now I have explored a bit. My test rig for R is in this form:
<hansl>
foreign language=R
fname <- <UTF8_filename>
file.info(fname)
fconv <- iconv(fname, from="utf-8", to=<TO_ENCODING>)
file.info(fconv)
end foreign
</hansl>
And here's what I find:
If <UTF8_filename> contains accented characters that are in my
Windows' locale codepage (for example c-cedilla), and <TO_ENCODING> is
just "" (empty string to indicate current locale), the second
invocation of file.info() works OK, though not surprsingly the first
one fails (actually, produces a bunch of NAs).
If <UTF8_filename> contains out-of-codepage characters (I tried
Greek), conversion to "" fails. Not surprising. In addition, iconv()
with to="UTF16" fails. It seems to me this last is an R bug. The
function iconvlist() shows that UTF16 is a valid target. The reported
cause of failure is that the "UTF-8" string contains an embedded NUL
character, which is impossible with valid UTF-8.
So I don't know what would happen if one were able to pass a UTF16
representation of the filename to a function such as file.info().
But I suspect R suffers from the problem that gretl had before 2018b,
namely that you can't access files that have out-of-codepage names
(and whose names must therefore be given in UTF16).
Allin
8:14 p.m.
Am 27.08.2018 um 21:42 schrieb Allin Cottrell:
And here's what I find:
If <UTF8_filename> contains accented characters that are in my
Windows' locale codepage (for example c-cedilla), and <TO_ENCODING> is
just "" (empty string to indicate current locale), the second
invocation of file.info() works OK, though not surprsingly the first
one fails (actually, produces a bunch of NAs).
So this case might be caught in
functions like gretl.loadmat which is
offered to R by gretl, right?
Or actually I guess I might mean the string gretl.dotdir, really.
If <UTF8_filename> contains out-of-codepage characters (I tried
Greek), conversion to "" fails. Not surprising. In addition, iconv()
with to="UTF16" fails. It seems to me this last is an R bug. The
function iconvlist() shows that UTF16 is a valid target.
Aha, interesting. I was puzzled by some of the failures in this area.
Where are R bugs reported, anyway? It turns out it's not only hard to
google for "R" stuff, but also for R bugs, because that's actually an R
package of that name...
One more thought, maybe "UTF-16LE" on little-endian machines?
But I suspect R suffers from the problem that gretl had before 2018b,
namely that you can't access files that have out-of-codepage names
(and whose names must therefore be given in UTF16).
Sounds plausible -- way to go, gretl!
thanks,
sven
9:27 p.m.
On Mon, 27 Aug 2018, Sven Schreiber wrote:
Am 27.08.2018 um 21:42 schrieb Allin Cottrell:
>
> And here's what I find:
>
> If <UTF8_filename> contains accented characters that are in my Windows'
> locale codepage (for example c-cedilla), and <TO_ENCODING> is just ""
> (empty string to indicate current locale), the second invocation of
> file.info() works OK, though not surprsingly the first one fails (actually,
> produces a bunch of NAs).
So this case might be caught in functions like gretl.loadmat which is offered
to R by gretl, right?
Or actually I guess I might mean the string gretl.dotdir, really.
Both gretl.dotdir and the matrix filename might contain non-ASCII
characters. There's now in git, but not yet snapshots, a
Windows-specific call to R's iconv() to convert the full filename
(dotdir + matrix-file) to the locale in gretl.loadmat. (Though
actually, perhaps gretl.dotdir should be converted in its own
right.) That, hopefully, should fix the case of non-ASCII filenames
that are representable in the locale.
> If <UTF8_filename> contains out-of-codepage characters (I
tried Greek),
> conversion to "" fails. Not surprising. In addition, iconv() with
> to="UTF16" fails. It seems to me this last is an R bug. The function
> iconvlist() shows that UTF16 is a valid target.
Aha, interesting. I was puzzled by some of the failures in this area. Where
are R bugs reported, anyway? It turns out it's not only hard to google for
"R" stuff, but also for R bugs, because that's actually an R package of
that
name...
"bugzilla" is the magic string to include with "R". But... you
should be very sure what you think might be an R bug really is an R
bug before reporting it, unless you want a roasting!
One more thought, maybe "UTF-16LE" on little-endian
machines?
I don't think that's the issue, but it should be checked.
> But I suspect R suffers from the problem that gretl had before
2018b,
> namely that you can't access files that have out-of-codepage names (and
> whose names must therefore be given in UTF16).
>
Sounds plausible -- way to go, gretl!
I did think of one other approach, namely having gretl recode the
script it's sending to R from UTF-8 to UTF-16 at source. That didn't
do any good; R barfed on the script from the start.
Allin
9:52 p.m.
On Mon, 27 Aug 2018, Allin Cottrell wrote:
[Y]ou should be very sure what you think might be an R bug really
is an R bug before reporting it, unless you want a roasting!
I'm getting closer to sure. My test involves the string "Anastasia"
spelled out in Greek letters. If I run this by itself through R's
iconv(string, from="utf-8", to="UTF16") it works fine. But if I put
any Roman letters into the string, as in "Anastasia/test" then it
fails, complaining "embedded nul in string". But of course both
UTF-8 and UTF-16 should have no trouble combining Greek and Roman
characters.
Allin
2:06 a.m.
On Mon, 27 Aug 2018, Allin Cottrell wrote:
On Mon, 27 Aug 2018, Allin Cottrell wrote:
> [Y]ou should be very sure what you think might be an R bug really is an R
> bug before reporting it, unless you want a roasting!
I'm getting closer to sure. My test involves the string "Anastasia" spelled
out in Greek letters. If I run this by itself through R's iconv(string,
from="utf-8", to="UTF16") it works fine. But if I put any Roman
letters into
the string, as in "Anastasia/test" then it fails, complaining "embedded
nul
in string". But of course both UTF-8 and UTF-16 should have no trouble
combining Greek and Roman characters.
Sorry to go on about this, but actually I now see why it _might_ not
be considered a bug. The UTF-16 sequence corresponding to
"Anastasia" in Greek letters contains no embedded nul byte, since
each of the Greek letters requires 2 non-empty bytes for its
representation. But the appended ASCII characters will each be
represented by a single "active" byte followed by a nul. (UTF-16
requires at least two bytes for each character, and pads with nuls
as needed.)
So I think what R's error message is trying to say is that the
result of conversion doesn't qualify as a string, where "string"
means a sequence of bytes _terminated_ by a nul byte. And true
enough, if you pass a third argument of 'toRaw=TRUE' iconv() gives
you the UTF-16 byte-array, including enbedded nuls, without
complaint. Now whether you can pass that array to an R function in
the role of a filename and expect anything useful to happen, I very
much doubt.
Allin
8:12 a.m.
Am 28.08.2018 um 04:06 schrieb Allin Cottrell:
Sorry to go on about this, but actually I now see why it _might_ not
be considered a bug. The UTF-16 sequence corresponding to "Anastasia"
in Greek letters contains no embedded nul byte, since each of the
Greek letters requires 2 non-empty bytes for its representation. But
the appended ASCII characters will each be represented by a single
"active" byte followed by a nul. (UTF-16 requires at least two bytes
for each character, and pads with nuls as needed.)
So I think what R's error message is trying to say is that the result
of conversion doesn't qualify as a string, where "string" means a
sequence of bytes _terminated_ by a nul byte.
But wouldn't that imply that R considers all UTF-16 strings as invalid
as long as there are some "simple" characters in there? If that's the
case, it would very much defeat the purpose of Unicode being a superset
of more restrictive encodings. So it still sounds like a bug, no?
cheers,
sven
6:18 p.m.
On Tue, 28 Aug 2018, Sven Schreiber wrote:
Am 28.08.2018 um 04:06 schrieb Allin Cottrell:
>
>
> Sorry to go on about this, but actually I now see why it _might_ not be
> considered a bug. The UTF-16 sequence corresponding to "Anastasia" in Greek
> letters contains no embedded nul byte, since each of the Greek letters
> requires 2 non-empty bytes for its representation. But the appended ASCII
> characters will each be represented by a single "active" byte followed by a
> nul. (UTF-16 requires at least two bytes for each character, and pads with
> nuls as needed.)
>
> So I think what R's error message is trying to say is that the result of
> conversion doesn't qualify as a string, where "string" means a sequence
of
> bytes _terminated_ by a nul byte.
But wouldn't that imply that R considers all UTF-16 strings as invalid as
long as there are some "simple" characters in there? If that's the case, it
would very much defeat the purpose of Unicode being a superset of more
restrictive encodings. So it still sounds like a bug, no?
Not really, IMO. No program can support "narrow" strings (ASCII, 8-bit
codepages, UTF-8) and "wide" ones (UTF-16, etc.) interchangeably.
I think R's error message is not very good, but the deal is basically,
"Look, we're not going to give you a UTF-16 byte array as if it were a
string, because we only do narrow strings. If you really want the
bytes, then use toRaw=TRUE and we'll give you them; what you do with
them is then up to you, but don't expect them to work where a string
is wanted."
Like gretl, R supports Unicode OK but only in the form of UTF-8. What
R apparently lacks is a mechanism to translate UTF-8 filenames to
UTF-16 on Windows automagically.
Allin
2061
days inactive
2062
days old
12 comments
2 participants
participants (2)
-
Allin Cottrell -
Sven Schreiber