On Fri, 17 May 2019, Riccardo (Jack) Lucchetti wrote: Answering myself: I put together a smal hansl function for doing just that: it should work on reasonably regular csv files. It's reported below (together with a small example): what do you guys think about adding it to the extra package? <hansl> set verbose off function string normalize_delimiters(string l) ret = l ret = regsub(ret, "[ ]*,[ ]*", "\t") ret = regsub(ret, "[ ]*;[ ]*", "\t") ret = regsub(ret, "[ ]+", "\t") return ret end function function matrix csv2mat(string filename) string s = readfile(filename) strings lines = strsplit(s, "\n") scalar r = nelem(lines) ### find number of columns l = normalize_delimiters(lines[1]) scalar c = strlen(regsub(l, "[^\t]", "")) + 1 matrix ret = zeros(r,c) i = 1 loop i = 1 ..r --quiet l = normalize_delimiters(lines[i]) ll = strsplit(l, "\t") loop j = 1 .. c --quiet ret[i,j] = atof(ll[j]) endloop endloop return ret end function r = 20 c = 4 delim = " , " test = floor(mnormal(r,c)) testfilename = "x.csv" outfile @testfilename --quiet loop i = 1 .. r --quiet loop j = 1 .. c-1 --quiet printf "%g%s", test[i, j], delim endloop printf "%g\n", test[i, c] endloop end outfile chk = readfile(testfilename) print s X = csv2mat(testfilename) print test X eval test-X <hansl> ------------------------------------------------------- Riccardo (Jack) Lucchetti Dipartimento di Scienze Economiche e Sociali (DiSES) Università Politecnica delle Marche (formerly known as Università di Ancona) r.lucchetti(a)univpm.it http://www2.econ.univpm.it/servizi/hpp/lucchetti -------------------------------------------------------

Am 19.05.2019 um 12:21 schrieb Riccardo (Jack) Lucchetti: Hi Jack, sorry for the delay, I know the feeling of being alone in a thread ;-) Coming back to your original post, indeed I think it would make sense to extend the built-in mread() function. In that sense it's not clear to me that putting something like this into extra would be the best solution. Related remark: Gretl already has a lot of heuristics of dealing with csv files, which is used by the 'open' command. So basically reading in a matrix from a csv file should be similar to doing: clear -> open thisfile.csv - matrix m = {dataset}. Of course you don't want to ditch the old dataset in place and start a new session, so this is not totally equivalent. But perhaps internally in an extended mread() function the existing backend C code could be reused. (BTW, my hunch is that you really have a dataset in a csv file and you want to sidestep gretl's dataset objects [series etc.] and instead want to go straight into a data-matrix context. Am I right?] cheers sven

This would be very nice of course, but in that case I would imagine the job would be less straightforward than it seems, because of the intrinsic differences between the eventual aims. When we read a csv file into a dataset, we need to have many more checks in place than when reading into a matrix. For example, we can just zap strings and substitute them win NAs; or the header, if present, can contain labels that wouldn'be allowed as indentifiers (eg empty). Or we could allow incoherent csv structures (like in the "malformed" example in my sample script). The reason why I brought this up in the first place is that I realised that a function like the one I coded in hansl would be very useful when translating code from Octave/Matlab; of course, in some cases you will want your script to operate on the same data as the matlab code, where reading from csv is customarily used in a much larger set of cases that reading data in. ------------------------------------------------------- Riccardo (Jack) Lucchetti Dipartimento di Scienze Economiche e Sociali (DiSES) Università Politecnica delle Marche (formerly known as Università di Ancona) r.lucchetti(a)univpm.it http://www2.econ.univpm.it/servizi/hpp/lucchetti -------------------------------------------------------

On Sun, 19 May 2019, Sven Schreiber wrote: A few points on this. 1) Jack's csv2mat is an outstanding example of accomplishing a lot with just a few lines of hansl. Of course this is not in the least unusual from Jack, but for the rest of us it's noteworthy all the same! 2) I take Jack's point that the "no error" criterion for reading a dataset from CSV (which we already do) is more restrictive than that for reading a matrix from CSV -- where we don't have to care about valid variable names, nor about handling non-numeric values, which we can just map onto "NA" without further ado. 3) Nonetheless, I find that it's not too difficult to handle the issues under point 2 in the context of our current CSV importation code. In current git, you can try out reading CSV into a matrix via mread() when the filename (or URL) has a ".csv" extension. Two comments on that: (a) "CSV" really just means delimited text, the delimiter doesn't have to be comma; and (b) if we want to pursue this option we could admit some other filename extensions. 4) One point supported by Jack's hansl code that is not supported by our built-in CSV importer is malformed CSV (e.g. some lines have more fields than others). I don't think we'd want to support this in our C code -- and actually I kinda wonder about the wisdom of supporting it at all. I'm attaching a sample script that derives from Jack's original upthread. It requires, and compares results with, Jack's csv2mat.inp. Allin

On Mon, 20 May 2019, Sven Schreiber wrote: Excellent, thanks a lot Allin, it works flawlessly on all the exaples I tried. Of course this makes my proposed addition to extra.gfn 100% moot, but I suppose the code in csv2mat.inp could be somehow reused as an example somewhere in the docs, where we show how to work with text files and strings... by th way: has anyone else but Francesca thought about the idea of splitting the "graphics" part from the Guide into a separate document? Ok with me. Well, the "sparse" package represents sparse matrices via a bundle, so we've got bread() and bwrite(). ------------------------------------------------------- Riccardo (Jack) Lucchetti Dipartimento di Scienze Economiche e Sociali (DiSES) Università Politecnica delle Marche (formerly known as Università di Ancona) r.lucchetti(a)univpm.it http://www2.econ.univpm.it/servizi/hpp/lucchetti -------------------------------------------------------