[Gretl-users] Datahub
Dear all,
Allin and I are working on automated data download from dbnomics (if you
don't know what I'm talking about: https://db.nomics.world). This led both
of us to using JSON object more systematically than we used to, and I have
to say JSON is very nice.
During my experiments, I wrote a little function for automatically
downloading datasets from DataHub (https://datahub.io). A small example
script follows:
<hansl>
set verbose off
function string datahub_get(string URL, string filename[null])
s = readfile(URL)
printf "reading %s...\n", URL
flush
printf "Dataset name: %s\n", jsonget(s, "$.title")
resources = jsonget(s, "$.resources..datahub.type")
got = 0
loop i = 1 .. nelem(resources) --quiet
string r = strstrip(jsonget(s, sprintf("$.resources[%d].datahub.type",
i-1)))
if r == "derived/csv"
hit = i-1
got = 1
break
endif
endloop
if got
if exists(filename)
ret = filename
else
fname = sprintf("%04d.csv", randgen1(i,0,9999))
if $windows
ret = sprintf("@dotdir\\%s", fname)
else
ret = sprintf("@dotdir/%s", fname)
endif
endif
printf "data URL found, reading...\n"
flush
path = jsonget(s, sprintf("$.resources[%d].path", hit))
c = readfile(path)
outfile "@ret" --quiet --write
print c
outfile --close
else
printf "No data found\n"
ret = ""
endif
return ret
end function
URL =
"https://pkgstore.datahub.io/core/pharmaceutical-drug-spending/19/datapackage.json"
fname = datahub_get(URL)
print fname
open "@fname"
</hansl>
Of course the function above can be enhanced in many ways, but the gist
is: once you have the URL to the dataset, the function will download the
data to a CSV file (you can supply a name if you want, otherwise a random
one will be generated for you). At that point, you may just open the CSV
file.
How do you get the URLs for the datasets you want? Easy:
* Navigate to the page of the dataset you want (for example,
https://datahub.io/core/co2-ppm)
* scroll to the bottom and copy the URL corresponding to the
"Datapackage.json" button. In most cases, it's just the page URL with
"/datapackage.json" appended at the end (for example,
"https://datahub.io/core/co2-ppm/datapackage.json")
* run the function datahub_get() with the URL string as first parameter;
you maty supply an optional second parameter if you want to save the CSV
file to a location of your choice.
Enjoy!
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------