5:23 p.m.
I'm wondering whether people would consider this function (which I'm
calling "strscrape" for now) worth having: it takes a string argument
and returns a row vector containing any numerical values encountered
in the string (or an empty matrix if none are found). There's a test
version in git but it's not documented; I'm not committed to keeping
it if doesn't attract much support.
Unlike using sscanf() to retrieve a matrix, the idea with this
function is that the numbers may be embedded in arbitrary text.
Numbers are identified as starting with either a minus sign (followed
directly by a digit) or a digit, and their length in characters is
decided by the C function strtod (so scientific notation is
acceptable). As the function stands, the decimal character is
unconditionally '.'.
One use for such a function would be "remedial" -- getting values back
from a gretl command when there's no relevant accessor. Here's a
semi-general illustration:
<hansl>
function matrix scrapevals (const string cmd, const list L[null])
string fname = sprintf("%s/_tmp.txt", $dotdir)
set force_decpoint on
outfile @fname --write --quiet
@cmd
outfile --close
string s = readfile(fname)
remove(fname)
return strscrape(s)
end function
open data4-10
list L = ENROLL CATHOL
matrix m = scrapevals("corr L --kendall", L)
print m
</hansl>
However, it could also be useful for processing text other than
gretl's own output. Here's a generic example:
<hansl>
string s="Here is some text 1.5e-4with embedded 12345 numbers \
stuck int0 1t."
matrix vals = strscrape(s)
print vals
</hansl>
which gives
vals (1 x 4)
0.00015000 12345. 0.0000 1.0000
Allin
8:39 p.m.
On Sun, 11 Dec 2016, Allin Cottrell wrote:
I'm wondering whether people would consider this function (which
I'm calling
"strscrape" for now) worth having: it takes a string argument and returns a
row vector containing any numerical values encountered in the string (or an
empty matrix if none are found). There's a test version in git but it's not
documented; I'm not committed to keeping it if doesn't attract much support.
I, for one, quite like it.
Unlike using sscanf() to retrieve a matrix, the idea with this
function is
that the numbers may be embedded in arbitrary text. Numbers are identified as
starting with either a minus sign (followed directly by a digit) or a digit,
and their length in characters is decided by the C function strtod (so
scientific notation is acceptable). As the function stands, the decimal
character is unconditionally '.'.
Now, how could I EVER object to THIS? ;)
Of course some rough edges will have to be dealt with. For example, the
output to
a = strscrape("1.2.3")
is, as of now,
[1.2, 3]
which of course makes sense but is only one of the possible conventions.
One use for such a function would be "remedial" -- getting
values back from a
gretl command when there's no relevant accessor. Here's a semi-general
illustration:
<hansl>
function matrix scrapevals (const string cmd, const list L[null])
string fname = sprintf("%s/_tmp.txt", $dotdir)
set force_decpoint on
outfile @fname --write --quiet
@cmd
outfile --close
string s = readfile(fname)
remove(fname)
return strscrape(s)
end function
open data4-10
list L = ENROLL CATHOL
matrix m = scrapevals("corr L --kendall", L)
print m
</hansl>
Actually, the function above would be even better if one used the --buffer
option to outfile, but I don't mean to be picky.
However, it could also be useful for processing text other than
gretl's own
output. Here's a generic example:
<hansl>
string s="Here is some text 1.5e-4with embedded 12345 numbers \
stuck int0 1t."
matrix vals = strscrape(s)
print vals
</hansl>
which gives
vals (1 x 4)
0.00015000 12345. 0.0000 1.0000
Very nice.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
11:10 p.m.
Em 12 de novembro de 2016, Jack escreveu:
On Sun, 11 Dec 2016, Allin Cottrell wrote:
I'm wondering whether people would consider this function (which I'm
> calling "strscrape" for now) worth having: it takes a string argument and
> returns a row vector containing any numerical values encountered in the
> string (or an empty matrix if none are found). There's a test version in
> git but it's not documented; I'm not committed to keeping it if doesn't
> attract much support.
>
I, for one, quite like it.
Me too. It could help apply web scrapping tasks using Gretl!
Unlike using sscanf() to retrieve a matrix, the idea with this
function is
> that the numbers may be embedded in arbitrary text. Numbers are identified
> as starting with either a minus sign (followed directly by a digit) or a
> digit, and their length in characters is decided by the C function strtod
> (so scientific notation is acceptable). As the function stands, the decimal
> character is unconditionally '.'.
>
Now, how could I EVER object to THIS? ;)
Of course some rough edges will have to be dealt with. For example, the
output to
a = strscrape("1.2.3")
is, as of now,
[1.2, 3]
which of course makes sense but is only one of the possible conventions.
To avoid that kind of situation I suggest to return the number as a array
of strings. So the user can explicit choose what to do. Furthermore, using
array of strings as the return value of the function can help users to
handle the "problem" of decimal comma.
One more thing: it would be great if we could have a function like this to
count the number of occurrences of a determined string. Something like this:
string S = "I like Gretl because Gretl is great and Gretl is complete"
scalar N = strscrapec(S, "Gretl") # strscrapc; String scrape counter
# result: N = 3
Best regards,
Henrique Andrade
11:45 p.m.
On Sun, 11 Dec 2016, Henrique Andrade wrote:
To avoid that kind of situation I suggest to return the number as a
array
of strings. So the user can explicit choose what to do. Furthermore, using
array of strings as the return value of the function can help users to
handle the "problem" of decimal comma.
Hmmm, no, sorry.
One more thing: it would be great if we could have a function like
this to
count the number of occurrences of a determined string. Something like this:
string S = "I like Gretl because Gretl is great and Gretl is complete"
scalar N = strscrapec(S, "Gretl") # strscrapc; String scrape counter
# result: N = 3
I don't like very much this either. If you need this sort of things
regularly, then use python. IMO hansl should stay focused on
statistics/econometrics. I can't really see a point in chasing python or
Julia on the data mining field. But I'm open to changing my mind.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
3:37 a.m.
Em 11 de dezembro de 2016, Jack escreveu:
On Sun, 11 Dec 2016, Henrique Andrade wrote:
> (...)
One more thing: it would be great if we could have a function like this to
> count the number of occurrences of a determined string. Something like
> this:
>
> string S = "I like Gretl because Gretl is great and Gretl is complete"
> scalar N = strscrapec(S, "Gretl") # strscrapc; String scrape counter
> # result: N = 3
>
I don't like very much this either. If you need this sort of things
regularly, then use python. IMO hansl should stay focused on
statistics/econometrics. I can't really see a point in chasing python or
Julia on the data mining field. But I'm open to changing my mind.
I'm a little bit confused now. Following your stream of thought, where is
the econometrics/statistics in the following code?
<hansl>
string s="Here is some text 1.5e-4with embedded 12345 numbers \
stuck int0 1t."
matrix vals = strscrape(s)
print vals
</hansl>
But it is ok, the question is just rhetorical. It was just a suggestion.
Best,
Henrique Andrade
12:47 a.m.
To avoid that kind of situation I suggest to return the number as a array of strings. So
the user can explicit choose what to do. Furthermore, using array of strings as the return
value of the function can help users to handle the "problem" of decimal comma.
This seems like adding an unnecessary layer of complexity. Why take a function
that pulls numbers out of a string, then put those numbers back into a string? Maybe
that'd be useful to some users in some contexts, but how generally applicable would
this be?
PS
2:21 p.m.
On Sun, 11 Dec 2016, Henrique Andrade wrote:
Em 12 de novembro de 2016, Jack escreveu:
>
> Of course some rough edges will have to be dealt with. For example, the
> output to
>
> a = strscrape("1.2.3")
>
> is, as of now,
>
> [1.2, 3]
>
> which of course makes sense but is only one of the possible conventions.
To avoid that kind of situation I suggest to return the number as a array
of strings. So the user can explicit choose what to do. Furthermore, using
array of strings as the return value of the function can help users to
handle the "problem" of decimal comma.
I don't really favor that. The decimal comma could be handled via an
optional second argument (boolean "comma", defaulting to false).
As for Jack's example, I think this can be improved by adding the
decimal separator to the list of candidate characters for starting a
number (provided it's immediately followed by a digit). Then
strscrape("1.2.3")
would return {1.2, 0.3}, which is, I think, fully defensible as a
left-to-right parsing.
One more thing: it would be great if we could have a function like
this to count the number of occurrences of a determined string.
Something like this:
string S = "I like Gretl because Gretl is great and Gretl is complete"
scalar N = strscrapec(S, "Gretl") # strscrapc; String scrape counter
# result: N = 3
That sort of thing is easily done with existing string functions:
<hansl>
function scalar strcount (string s, string p)
scalar n = 0
scalar plen = strlen(p)
loop while strlen(s) > 0 -q
s = strstr(s, p)
if strlen(s) > 0
n++
s += plen
endif
endloop
return n
end function
string s = "I like Gretl because Gretl is great and Gretl is complete"
eval strcount(s, "Gretl")
</hansl>
The scraping of numerical values, on the other hand, could not be done
(or at least, not at all conveniently) in hansl without additional
low-level functions -- equivalents of the C functions strcspn, strtod
and isdigit. We _could_ add those if we thought it worthwhile.
Allin
2936
days inactive
2937
days old
6 comments
4 participants
participants (4)
-
Allin Cottrell -
Henrique Andrade -
Riccardo (Jack) Lucchetti -
Summers, Peter