On Tue, 15 Oct 2013, Sven Schreiber wrote: Can you give some more detail? The following works OK for me: <hansl> open monthly1.txt store monthly1.gdt open monthly2.txt store monthly2.gdt clear open monthly1.gdt append monthly2.gdt print -o </hansl> where monthly1.txt is obs,x1,y1 2005:01,1,21 2005:02,2,22 2005:03,3,23 2005:04,4,24 2005:05,5,25 2005:06,6,26 2005:07,7,27 2005:08,8,28 2005:09,9,29 2005:10,10,30 2005:11,11,31 2005:12,12,32 and monthly2.txt is obs,x2,y2 2005:08,1,21 2005:09,2,22 2005:10,3,23 2005:11,4,24 2005:12,5,25 2006:01,6,26 2006:02,7,27 2006:03,8,28 2006:04,9,29 2006:05,10,30 2006:06,11,31 2006:07,12,32 (different variables, overlapping ranges). The output I get is: x y x2 y2 2005:01 1 21 2005:02 2 22 2005:03 3 23 2005:04 4 24 2005:05 5 25 2005:06 6 26 2005:07 7 27 2005:08 8 28 1 21 2005:09 9 29 2 22 2005:10 10 30 3 23 2005:11 11 31 4 24 2005:12 12 32 5 25 2006:01 6 26 2006:02 7 27 2006:03 8 28 2006:04 9 29 2006:05 10 30 2006:06 11 31 2006:07 12 32 Allin

On Wed, 16 Oct 2013, Sven Schreiber wrote: [...] Bingo! Something has come unstuck with appending data via the menus. This must be fallout from the change I made to the Open data menu action recently. Should be an easy fix, I think. Allin

I read in the gretl changelog this "readfile() function: check for valid UTF-8 and recode if possible if the text is not UTF-8" But I see this is not properly working. I tried to open the attached file (ISO-8859) but in running the 'readline' command I obtain the error message "Invalid byte sequence in conversion input" -- Firma Arista *Ignacio Díaz-Emparanza* Zuzendaria/Director ignacio.diaz-emparanza(a)ehu.es <mailto:ignacio.diaz.emparanza@ehu.es> 94 6013732 *EKONOMIA APLIKATUA III SAILA (EKONOMETRIA ETA ESTATISTIKA)/ DEPARTAMENTO DE ECONOMÍA APLICADA III (ECONOMETRÍA Y ESTADÍSTICA) UPV/EHU* Avda. Lehendakari Aguirre, 83 | 48015 BILBAO *T.: +34 946013740* | *F.: +34 946013754* *www.ea3.ehu.es* <http://www.ea3.ehu.es> ERNE! Baliteke mezu honen zatiren bat edo mezu osoa legez babestuta egotea. Mezuak badu bere hartzailea. Okerreko helbidera heldu bada (helbidea gaizki idatzi, transmisioak huts egin) eman abisu igorleari, korreo honi erantzunda. Kontuz! Mezua ez bada zuretzat, ez erabili, ez zabaldu beste inori, ez kopiatu eta ez baliatu. ¡ATENCIÓN! Este mensaje contiene información privilegiada o confidencial a la que sólo tiene derecho a acceder el destinatario. Si usted lo recibe por error le agradeceríamos que no hiciera uso de la información y que se pusiese en contacto con el remitente. E-mail hau inprimatu baino lehen egiaztatu inprimatzeko beharra. Antes de imprimir este e-mail piense bien si es necesario hacerlo.

On 18/10/13 13:05, Hélio Guilherme wrote: Thanks, Helio. I normally use the shell command 'recode' to convert files between different charsets. The problem is that sometimes this is not possible. In fact I was trying to process a web page with charset ISO-8859 to produce this table (after some modifications, obviously) with string s = readfile("http://www.elmundo.es/elmundodeporte/futbol/liga-primera/clasificacion.html") but this produces the same error. -- Firma Arista *Ignacio Díaz-Emparanza* Zuzendaria/Director ignacio.diaz-emparanza(a)ehu.es <mailto:ignacio.diaz.emparanza@ehu.es> 94 6013732 *EKONOMIA APLIKATUA III SAILA (EKONOMETRIA ETA ESTATISTIKA)/ DEPARTAMENTO DE ECONOMÍA APLICADA III (ECONOMETRÍA Y ESTADÍSTICA) UPV/EHU* Avda. Lehendakari Aguirre, 83 | 48015 BILBAO *T.: +34 946013740* | *F.: +34 946013754* *www.ea3.ehu.es* <http://www.ea3.ehu.es> ERNE! Baliteke mezu honen zatiren bat edo mezu osoa legez babestuta egotea. Mezuak badu bere hartzailea. Okerreko helbidera heldu bada (helbidea gaizki idatzi, transmisioak huts egin) eman abisu igorleari, korreo honi erantzunda. Kontuz! Mezua ez bada zuretzat, ez erabili, ez zabaldu beste inori, ez kopiatu eta ez baliatu. ¡ATENCIÓN! Este mensaje contiene información privilegiada o confidencial a la que sólo tiene derecho a acceder el destinatario. Si usted lo recibe por error le agradeceríamos que no hiciera uso de la información y que se pusiese en contacto con el remitente. E-mail hau inprimatu baino lehen egiaztatu inprimatzeko beharra. Antes de imprimir este e-mail piense bien si es necesario hacerlo.

On Fri, 18 Oct 2013, Ignacio Diaz-Emparanza wrote: The question is, what codeset are we to assume we're converting from? To date we have assumed that if text is not UTF-8 it will be in the encoding of the current locale, and have used the GLib function g_locale_to_utf8() on the imported text. This will fail if your locale codeset is in fact UTF-8, which I presume is what's happening in your case. I've now made this a little smarter in CVS. First we check if the current locale codeset is UTF-8. If so, we avoid using g_locale_to_utf8() and instead use g_convert(), guessing at ISO-8859-15 as the source codeset. If the current locale codeset is _not_ UTF-8 we try using that as the source encoding; but if that fails we try ISO-8859-15. Perhaps we should offer an optional second argument to readfile(), allowing the user to specify the source codeset. By the way, the correspondence on this topic illustrates how tricky this whole business is. Ignacio attached a file which he said was in ISO-8859, but in fact what came across via email was an ASCII file with question marks in place of accented characters. Helio gave an inline example of a file that was again supposed to be in ISO-8859-15, but what came across via email here was in fact UTF-8. If you want to illustrate anything to do with codesets via email, it's necessary to zip or tar the files in question; otherwise you have no idea what your reader is going to see! Allin

On Fri, 18 Oct 2013, Ignacio Diaz-Emparanza wrote: You're not the only one :( ------------------------------------------------------- Riccardo (Jack) Lucchetti Dipartimento di Scienze Economiche e Sociali (DiSES) Università Politecnica delle Marche (formerly known as Università di Ancona) r.lucchetti(a)univpm.it http://www2.econ.univpm.it/servizi/hpp/lucchetti -------------------------------------------------------

On 18/10/13 18:40, Allin Cottrell wrote: Thanks ! With respect to the 'open' (importing CSV) command I think we may leave the responsability of using a correct UTF8 codeset to the user, but probably the error message that emerges in trying to import from an incorrect codeset could be more explicit. With the table I sent you, the error I obtain is <output> Binary data (225) encountered (line 9:4): this is not a valid text file </output> I assume the program in this conditions cannot distinguish an accent or symbol of a non-UTF8 codeset from another binary element, so a solution may be to add a Warning (after the error message) such as "Warning: Please note that gretl uses UTF8 codesetfor importing plaintext or CSV data." Appart from that, I am seeing that in my table the first accented character is at line 10, position 5, so I think the information given in the error message (line 9:4) is incorrect. -- Firma Arista *Ignacio Díaz-Emparanza* Zuzendaria/Director ignacio.diaz-emparanza(a)ehu.es <mailto:ignacio.diaz.emparanza@ehu.es> 94 6013732 *EKONOMIA APLIKATUA III SAILA (EKONOMETRIA ETA ESTATISTIKA)/ DEPARTAMENTO DE ECONOMÍA APLICADA III (ECONOMETRÍA Y ESTADÍSTICA) UPV/EHU* Avda. Lehendakari Aguirre, 83 | 48015 BILBAO *T.: +34 946013740* | *F.: +34 946013754* *www.ea3.ehu.es* <http://www.ea3.ehu.es> ERNE! Baliteke mezu honen zatiren bat edo mezu osoa legez babestuta egotea. Mezuak badu bere hartzailea. Okerreko helbidera heldu bada (helbidea gaizki idatzi, transmisioak huts egin) eman abisu igorleari, korreo honi erantzunda. Kontuz! Mezua ez bada zuretzat, ez erabili, ez zabaldu beste inori, ez kopiatu eta ez baliatu. ¡ATENCIÓN! Este mensaje contiene información privilegiada o confidencial a la que sólo tiene derecho a acceder el destinatario. Si usted lo recibe por error le agradeceríamos que no hiciera uso de la información y que se pusiese en contacto con el remitente. E-mail hau inprimatu baino lehen egiaztatu inprimatzeko beharra. Antes de imprimir este e-mail piense bien si es necesario hacerlo.