10:32 p.m.
Hi all,
I thought this could be of interest to some of you. Gretl has the
built-in functions jsonget() and jsongetb() which is very nice!
I wasn't aware about json-files until recently. At work we use these
files (apart from yaml-files) as definition files defining specific
parameters or variables of interest globally.
Imagine you have a model framework and you don't want to change any
parameter in your generically written code as it just works and any
further change may break it. Instead, you JUST pass json-files to the
framework which include all relevant information such as the list of
variables and eventual manipulations of these, specific settings on the
sample, choice of the estimator etc. You can easily put all these things
into json-files.
I've uploaded an example on how to use this with gretl here:
https://github.com/atecon/json_gretl
Best,
Artur
3:21 p.m.
Am 26.02.2019 um 23:32 schrieb Artur T.:
I've uploaded an example on how to use this with gretl here:
https://github.com/atecon/json_gretl
To be honest, apart from having a well defined exchange format when
several programming languages are mixed I don't understand the value
added over storing what you need in a bundle (which is gretl-only,
obviously).
Talking about mixing programming languages, maybe some of the json stuff
could be leveraged for gretl's 'foreign' interface. What I mean is an
easier or more flexible way to pass stuff from gretl/hansl to the
foreign language (Python, Julia, etc.).
cheers,
sven
10:19 p.m.
Am 27.02.19 um 16:21 schrieb Sven Schreiber:
Am 26.02.2019 um 23:32 schrieb Artur T.:
> I've uploaded an example on how to use this with gretl here:
> https://github.com/atecon/json_gretl
To be honest, apart from having a well defined exchange format when
several programming languages are mixed I don't understand the value
added over storing what you need in a bundle (which is gretl-only,
obviously).
Fair enough, for pure gretl purpose, a bundle would be the natural
alternative. But as you said, the json may be also used for other
software packages ensuring one has the same configuration.
Talking about mixing programming languages, maybe some of the json
stuff
could be leveraged for gretl's 'foreign' interface. What I mean is an
easier or more flexible way to pass stuff from gretl/hansl to the
foreign language (Python, Julia, etc.).
Very good idea! For instance, for R and Python there exist json
read/write functions. Guess that would be an interesting project for future.
Best,
Artur
7:23 a.m.
Am 27.02.2019 um 23:19 schrieb Artur T.:
Am 27.02.19 um 16:21 schrieb Sven Schreiber:
> Talking about mixing programming languages, maybe some of the
json
> stuff could be leveraged for gretl's 'foreign' interface. What I mean
> is an easier or more flexible way to pass stuff from gretl/hansl to
> the foreign language (Python, Julia, etc.).
Very good idea! For instance, for R and Python there exist json
read/write functions. Guess that would be an interesting project for
future.
I'm still talking about stuff I don't fully understand, but maybe we
should have a hansl function that writes out the contents of a bundle
(or let's say, a well-defined subset of possible contents) as a json
string or file. That would complement the existing native jsongetb
function.
Perhaps such a thing could go into the extra.gfn add-on.
cheers
sven
7:43 a.m.
On Thu, 28 Feb 2019, Sven Schreiber wrote:
Am 27.02.2019 um 23:19 schrieb Artur T.:
> Am 27.02.19 um 16:21 schrieb Sven Schreiber:
>> Talking about mixing programming languages, maybe some of the json stuff
>> could be leveraged for gretl's 'foreign' interface. What I mean is an
>> easier or more flexible way to pass stuff from gretl/hansl to the foreign
>> language (Python, Julia, etc.).
>
> Very good idea! For instance, for R and Python there exist json read/write
> functions. Guess that would be an interesting project for future.
I'm still talking about stuff I don't fully understand, but maybe we should
have a hansl function that writes out the contents of a bundle (or let's say,
a well-defined subset of possible contents) as a json string or file. That
would complement the existing native jsongetb function.
Perhaps such a thing could go into the extra.gfn add-on.
Hm, that'd be quite complex to write, but it's a good idea.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
9:50 a.m.
Am 28.02.2019 um 08:43 schrieb Riccardo (Jack) Lucchetti:
On Thu, 28 Feb 2019, Sven Schreiber wrote:
> I'm still talking about stuff I don't fully understand,
but maybe we
> should have a hansl function that writes out the contents of a bundle
> (or let's say, a well-defined subset of possible contents) as a json
> string or file. That would complement the existing native jsongetb
> function.
> Perhaps such a thing could go into the extra.gfn add-on.
Hm, that'd be quite complex to write, but it's a good idea.
He, is that a challenge? ;-)
As I said, my json knowledge is limited to the Wikipedia entry, but does
the following prototype give a valid json document?
cheers
sven
<hansl>
function string matrowcommasep(matrix m)
# m should be a row vector
m = vec(m)'
string mp = strstrip( sprintf("%20.10g", m) )
# remove multiple blanks
# (this could be done cleverly with regsub)
loop while instring(mp, " ") --quiet # two blanks
mp = strsub(mp, " ", " ") # one blank
endloop
# comma as separator
return strsub(mp, " ", ",")
end function
function string bundle2json(bundle b)
string out = sprintf("{\n")
strings keys = getkeys(b)
loop i = 1..nelem(keys) --quiet
string key = keys[i]
if typeof(b[key]) == 1 # scalar
out ~= sprintf("\"%s\": %g,\n", key, b[key])
elif typeof(b[key]) == 3 # matrix
out ~= sprintf("\"%s\": [", key)
loop r = 1..rows(b[key]) --quiet
out ~= sprintf("\n[") ~ matrowcommasep(b[key][r,]) ~ "],"
endloop
# strip the trailing comma and close array
out = out[1: strlen(out)-1] ~ sprintf("\n],\n")
else
funcerr "type not implemented"
endif
endloop
# replace trailing comma and newline
return out[1: strlen(out) - 2] ~ sprintf("\n}")
end function
#### test code
bundle b = defbundle("mymat",I(2), "myscal",3)
eval bundle2json(b)
</hansl>
10:10 a.m.
On Thu, 28 Feb 2019, Sven Schreiber wrote:
Am 28.02.2019 um 08:43 schrieb Riccardo (Jack) Lucchetti:
> On Thu, 28 Feb 2019, Sven Schreiber wrote:
>> I'm still talking about stuff I don't fully understand, but maybe we
>> should have a hansl function that writes out the contents of a bundle (or
>> let's say, a well-defined subset of possible contents) as a json string or
>> file. That would complement the existing native jsongetb function.
>> Perhaps such a thing could go into the extra.gfn add-on.
>
> Hm, that'd be quite complex to write, but it's a good idea.
He, is that a challenge? ;-)
Yes. Also known as "bait" ;-)
As I said, my json knowledge is limited to the Wikipedia entry, but
does the
following prototype give a valid json document?
It is. JSON documents normally don't contain newlines. The complexity I
was referring to is linked to the myriad of data types we have in hansl,
each of which requires a different JSON-isation. For example, I would
have stored matrices in a completely different way (that is, vectorisation
+ 2 scalars for dimensions, similar to what we use in mread/mwrite), but
your solution is also nice and has its own advantages.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
11:28 a.m.
Am 28.02.2019 um 11:10 schrieb Riccardo (Jack) Lucchetti:
On Thu, 28 Feb 2019, Sven Schreiber wrote:
> As I said, my json knowledge is limited to the Wikipedia entry, but
> does the following prototype give a valid json document?
It is. JSON documents normally don't contain newlines. The complexity
I was referring to is linked to the myriad of data types we have in
hansl, each of which requires a different JSON-isation. For example, I
would have stored matrices in a completely different way (that is,
vectorisation + 2 scalars for dimensions, similar to what we use in
mread/mwrite), but your solution is also nice and has its own advantages.
I guess one would try to target a format that is easily understood by
the already existing json tools in R, Python etc. Maybe Artur knows what
they would look like.
cheers
sven
6:03 p.m.
On Thu, 28 Feb 2019, Sven Schreiber wrote:
Am 27.02.2019 um 23:19 schrieb Artur T.:
> Am 27.02.19 um 16:21 schrieb Sven Schreiber:
>> Talking about mixing programming languages, maybe some of the json stuff
>> could be leveraged for gretl's 'foreign' interface. What I mean is an
>> easier or more flexible way to pass stuff from gretl/hansl to the foreign
>> language (Python, Julia, etc.).
>
> Very good idea! For instance, for R and Python there exist json read/write
> functions. Guess that would be an interesting project for future.
I'm still talking about stuff I don't fully understand, but maybe we should
have a hansl function that writes out the contents of a bundle (or let's say,
a well-defined subset of possible contents) as a json string or file.
bwrite() already serializes a bundle as XML. Is there a substantial
advantage in having a JSON serialization too? If so, is that best done
by writing a new serialization, or by transcoding the XML? (Googling
"convert XML to JSON" shows up tons of transcoding options.)
Allin
9:05 p.m.
Am 28.02.2019 um 19:03 schrieb Allin Cottrell:
On Thu, 28 Feb 2019, Sven Schreiber wrote:
> I'm still talking about stuff I don't fully understand,
but maybe we
> should have a hansl function that writes out the contents of a bundle
> (or let's say, a well-defined subset of possible contents) as a json
> string or file.
bwrite() already serializes a bundle as XML. Is there a substantial
advantage in having a JSON serialization too? If so, is that best done
by writing a new serialization, or by transcoding the XML? (Googling
"convert XML to JSON" shows up tons of transcoding options.)
Artur can perhaps give a better answer, but my impression with xml is
that it is too generic. You can do a lot of stuff with it, but in the
end that requires to have some clever xslt or similar thing which for me
basically means the same amount of work as simply exporting it in the
desired format in the first place.
So what's the use of a gretl bundle written out in xml from, say,
Python's point of view? Well, given the nice xml tools for Python I can
imagine it's not too difficult to get a tailor-made solution to grab the
matrices in there. So yes, maybe the json thing isn't needed. For target
languages without such tools my first impression is that reading and
processing json is simpler.
FWIW,
sven
10:14 p.m.
On Thu, 28 Feb 2019, Sven Schreiber wrote:
Am 28.02.2019 um 19:03 schrieb Allin Cottrell:
> On Thu, 28 Feb 2019, Sven Schreiber wrote:
>> I'm still talking about stuff I don't fully understand, but maybe we
>> should have a hansl function that writes out the contents of a bundle (or
>> let's say, a well-defined subset of possible contents) as a json string or
>> file.
>
> bwrite() already serializes a bundle as XML. Is there a substantial
> advantage in having a JSON serialization too? If so, is that best done by
> writing a new serialization, or by transcoding the XML? (Googling "convert
> XML to JSON" shows up tons of transcoding options.)
Artur can perhaps give a better answer, but my impression with xml is that it
is too generic. You can do a lot of stuff with it, but in the end that
requires to have some clever xslt or similar thing which for me basically
means the same amount of work as simply exporting it in the desired format in
the first place.
So what's the use of a gretl bundle written out in xml from, say, Python's
point of view? Well, given the nice xml tools for Python I can imagine it's
not too difficult to get a tailor-made solution to grab the matrices in
there. So yes, maybe the json thing isn't needed.
Interesting ideas all round, but before pursuing them it would seem
to be worth figuring out: What are the actual targets? (That is, the
other software to which we'd like to be able to pass gretl bundles.)
And what can they (best) parse? Can they happily read JSON but not
XML, or vice versa?
For target languages without such tools my first impression is
that reading and processing json is simpler.
The relevant tools are readily available in the free software domain
(e.g. libxml2 and json-glib). But if target software S is "without
such tools" (i.e. doesn't link to such libraries and expose their
functionality natively, as with gretl's jsonget and xmlget) it would
probably be a rather massive job to implement them in S's language.
IMO, S would have to offer some very special value-added to make
this worthwhile.
Allin
8:58 a.m.
Am 28.02.2019 um 23:14 schrieb Allin Cottrell
Interesting ideas all round, but before pursuing them it would seem
to
be worth figuring out: What are the actual targets? (That is, the other
software to which we'd like to be able to pass gretl bundles.) And what
can they (best) parse? Can they happily read JSON but not XML, or vice
versa?
[I'm tempted to move this to the development list, but becaue this is on
a meta level maybe it is still interesting for general users.]
True. Again, Artur seems to have a concrete use case, but apart from
that maybe it's actually better to address a language-independent free
standard instead of various specific targets. What about HDF5?
cheers
sven
12:07 p.m.
Sorry, for the late reply. I was away for a couple of days.
Am 28.02.19 um 23:14 schrieb Allin Cottrell:
On Thu, 28 Feb 2019, Sven Schreiber wrote:
> Am 28.02.2019 um 19:03 schrieb Allin Cottrell:
>> On Thu, 28 Feb 2019, Sven Schreiber wrote:
>
>>> I'm still talking about stuff I don't fully understand, but maybe we
>>> should have a hansl function that writes out the contents of a
>>> bundle (or let's say, a well-defined subset of possible contents) as
>>> a json string or file.
>>
>> bwrite() already serializes a bundle as XML. Is there a substantial
>> advantage in having a JSON serialization too? If so, is that best
>> done by writing a new serialization, or by transcoding the XML?
>> (Googling "convert XML to JSON" shows up tons of transcoding options.)
>
> Artur can perhaps give a better answer, but my impression with xml is
> that it is too generic. You can do a lot of stuff with it, but in the
> end that requires to have some clever xslt or similar thing which for
> me basically means the same amount of work as simply exporting it in
> the desired format in the first place.
>
> So what's the use of a gretl bundle written out in xml from, say,
> Python's point of view? Well, given the nice xml tools for Python I
> can imagine it's not too difficult to get a tailor-made solution to
> grab the matrices in there. So yes, maybe the json thing isn't needed.
Interesting ideas all round, but before pursuing them it would seem to
be worth figuring out: What are the actual targets? (That is, the other
software to which we'd like to be able to pass gretl bundles.) And what
can they (best) parse? Can they happily read JSON but not XML, or vice
versa?
As said here, my original post was just to show how to use a
JSON-configuration file which is readable by gretl and other software
packages such as Python, R and Julia. At work, JSON (as well as even
simple yaml-files) files are mainly used as configuration files having a
very simple structure as I have tried to show in my original post. Sven
then had the idea of using JSON as an exchange format.
I think Allin is right when asking for what purpose one wants to
implement these ideas. When working with the "foreign-block" interacting
and exchanging objects with other software packages, XML may be the
right format from gretl's perspective as gretl already has the bread()
and bwrite() functions for such purposes.
And as already mentioned here, there exist read/write XML packages for
all popular software packages such as R, Python and Julia (the packages
I would personally focus on). Also, transformation from XML to JSON and
vice versa can be become quite costly I guess -- if possible let's avoid
this.
Also, if I am not totally wrong, gretl's XML handling covers more data
types --- actually all which can be put into a gretl bundle --- compared
to what can be handled with jsonget(). The help file only lists the
handling of: double (floating-point), int (integer) and strings.
The relevant tools are readily available in the free software domain
(e.g. libxml2 and json-glib). But if target software S is "without such
tools" (i.e. doesn't link to such libraries and expose their
functionality natively, as with gretl's jsonget and xmlget) it would
probably be a rather massive job to implement them in S's language. IMO,
S would have to offer some very special value-added to make this
worthwhile.
I would say that if language S has not implemented such freely available
tools, it's not "our" task to write a library for S. In this case,
interaction is strictly limited to the exchange of simple ascii files.
Best,
Artur
2133
days inactive
2137
days old
12 comments
4 participants
participants (4)
-
Allin Cottrell -
Artur T. -
Riccardo (Jack) Lucchetti -
Sven Schreiber