1:36 p.m.
Hello
I refer to to an old problem. Requesting a bundle of data from
dbnomics database after the versions "dbnomics 2.0" is very slow. It
seem to be correlated to the reason that all metadata of the times
series are transmitted and not only the values. With a newer version of
the api v22 it is possible to eleminate the metatdata. See the mail
below I have got from the folks of dbnomics.
Is there the the possibility to change the "dbnomics 0.39" in that way
which comes with the version 2021b of gretl for MS-Windows?
Regards
Klaus
bruno.duye <https://forum.db.nomics.world/u/bruno.duye>
March 7
Hi @Klaus <https://forum.db.nomics.world/u/klaus>;, we added a |metadata|
argument that can be set to false to disable metadata.
It is documented in API doc: http://api.db.nomics.world/v22/apidocs
------------------------------------------------------------------------
2:52 p.m.
On Mon, 8 Mar 2021, klaus.hasenbach(a)web.de wrote:
Hello
I refer to to an old problem. Requesting a bundle of data from
dbnomics database after the versions "dbnomics 2.0" is very slow. It
seem to be correlated to the reason that all metadata of the times
series are transmitted and not only the values. With a newer version of
the api v22 it is possible to eleminate the metatdata. See the mail
below I have got from the folks of dbnomics.
Is there the the possibility to change the "dbnomics 0.39" in that way
which comes with the version 2021b of gretl for MS-Windows?
Yes. That's good news! For a time under the v22 API there was no way
to decline getting all the metadata.
Allin
12:59 p.m.
On Mon, 8 Mar 2021, klaus.hasenbach(a)web.de wrote:
I refer to to an old problem. Requesting a bundle of data from
dbnomics database after the versions "dbnomics 2.0" is very slow. It
seem to be correlated to the reason that all metadata of the times
series are transmitted and not only the values. With a newer version of
the api v22 it is possible to eleminate the metatdata. See the mail
below I have got from the folks of dbnomics.
Is there the the possibility to change the "dbnomics 0.39" in that way
which comes with the version 2021b of gretl for MS-Windows?
The dbnomics addon is now updated to version 0.40, which eliminates
the excessive metadata download. Thanks for drawing the new API
option to our attention.
Allin
1:38 a.m.
Hello Allin,
"metada" are set to "false"according the new dbnomics.gfn version
4.0.
Thank you for doing so. But running it is not faster than the older
version. Looking for reasons: The decline of the metadata does no work
under MS- Windows? Or do I have to write my bundl comands different?
Klaus
Am 08.03.2021 um 15:52 schrieb Allin Cottrell:
On Mon, 8 Mar 2021, klaus.hasenbach(a)web.de wrote:
> Hello
>
> I refer to to an old problem. Requesting a bundle of data from
> dbnomics database after the versions "dbnomics 2.0" is very slow. It
> seem to be correlated to the reason that all metadata of the times
> series are transmitted and not only the values. With a newer version of
> the api v22 it is possible to eleminate the metatdata. See the mail
> below I have got from the folks of dbnomics.
>
> Is there the the possibility to change the "dbnomics 0.39" in that way
> which comes with the version 2021b of gretl for MS-Windows?
Yes. That's good news! For a time under the v22 API there was no way
to decline getting all the metadata.
Allin
_______________________________________________
Gretl-users mailing list -- gretl-users(a)gretlml.univpm.it
To unsubscribe send an email to gretl-users-leave(a)gretlml.univpm.it
Website:
https://gretlml.univpm.it/postorius/lists/gretl-users.gretlml.univpm.it/
12:13 p.m.
Thank you, Allin!
But I guess there is a bug in dbnonmics 0.4 with Windows 10. Bundle
requests via mask are slow as before (see for examble the attached scipt).
And single requests such as
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z are aborted immediatly with
error:
"Keine Datenbank wurde geöffnet"=no databank was opened
Both request are running very well with high speed and no error under
dbnomics 0.2
Klaus
Am 15.03.2021 um 13:59 schrieb Allin Cottrell:
On Mon, 8 Mar 2021, klaus.hasenbach(a)web.de wrote:
> I refer to to an old problem. Requesting a bundle of data from
> dbnomics database after the versions "dbnomics 2.0" is very slow. It
> seem to be correlated to the reason that all metadata of the times
> series are transmitted and not only the values. With a newer version of
> the api v22 it is possible to eleminate the metatdata. See the mail
> below I have got from the folks of dbnomics.
>
> Is there the the possibility to change the "dbnomics 0.39" in that way
> which comes with the version 2021b of gretl for MS-Windows?
The dbnomics addon is now updated to version 0.40, which eliminates
the excessive metadata download. Thanks for drawing the new API option
to our attention.
Allin
_______________________________________________
Gretl-users mailing list -- gretl-users(a)gretlml.univpm.it
To unsubscribe send an email to gretl-users-leave(a)gretlml.univpm.it
Website:
https://gretlml.univpm.it/postorius/lists/gretl-users.gretlml.univpm.it/
1:38 p.m.
On Mon, 29 Mar 2021, klaus.hasenbach(a)web.de wrote:
Thank you, Allin!
But I guess there is a bug in dbnonmics 0.4 with Windows 10. Bundle
requests via mask are slow as before (see for examble the attached scipt).
And single requests such as
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z are aborted immediatly with
error:
"Keine Datenbank wurde geöffnet"=no databank was opened
Both request are running very well with high speed and no error under
dbnomics 0.2
Thanks, Klaus. I'll look into this.
Allin
9:56 p.m.
On Mon, 29 Mar 2021, klaus.hasenbach(a)web.de wrote:
Thank you, Allin!
But I guess there is a bug in dbnonmics 0.4 with Windows 10. Bundle
requests via mask are slow as before (see for examble the attached scipt).
I see what you mean. Running a slightly cut-down version of your
multi-series request, with mask "DEU+GBR+IAA..Q" and a maximum of
250 series, I'm seeing timings and JSON sizes as follows:
dbnomics 0.4, with metadata: 0m56.569s, 2317360 bytes
dbnomics 0.4, no metadata: 0m55.649s, 2296071 bytes
So cutting out the metadata makes little difference to the time or
size. (Up till now we haven't been setting metadata=0 for this sort
of request -- we missed this case -- but it hardly helps.)
If I invoke the dbnomics v21 API instead, I get this:
time 0m4.886s, size 1905768 bytes
I still have to determine what's taking 10 times as long: dbnomics
or gretl's processing of the data. But at this point my suspicion is
that it's dbnomics.
And single requests such as
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z are aborted immediatly
with
error:
"Keine Datenbank wurde geöffnet"=no databank was opened
I'm not able to replicate that. I tried on Windows 10 today and the
series arrived immediately.
Allin
11:28 p.m.
On Mon, 29 Mar 2021, Allin Cottrell wrote:
On Mon, 29 Mar 2021, klaus.hasenbach(a)web.de wrote:
> Thank you, Allin!
>
> But I guess there is a bug in dbnonmics 0.4 with Windows 10. Bundle
> requests via mask are slow as before (see for examble the attached scipt).
I see what you mean. Running a slightly cut-down version of your multi-series
request, with mask "DEU+GBR+IAA..Q" and a maximum of 250 series, I'm seeing
timings and JSON sizes as follows:
dbnomics 0.4, with metadata: 0m56.569s, 2317360 bytes
dbnomics 0.4, no metadata: 0m55.649s, 2296071 bytes
So cutting out the metadata makes little difference to the time or
size. (Up till now we haven't been setting metadata=0 for this
sort of request -- we missed this case -- but it hardly helps.)
If I invoke the dbnomics v21 API instead, I get this:
time 0m4.886s, size 1905768 bytes
I still have to determine what's taking 10 times as long: dbnomics
or gretl's processing of the data. But at this point my suspicion
is that it's dbnomics.
I was wrong. The retrieval time is very similar across the v21 and
v22 APIs. What was taking so long was something that I believe is
now quite unnecessary, namely "fixing" the dataset "dimensions"
objects in the retrieved JSON. I think we were doing this because
(a) it seemed to us that the v22 API was kinda broken at one point
and/or (b) that gretl didn't support nested arrays and so some
complex translation was needed. (We now have nested arrays.)
Anyway, I'll put out version 0.41 of the dbnomics addon shortly,
with the "fix dimensions" code de-activated. It would be good if
people can check for any breakage that may result!
Allin
9:58 a.m.
Thank`s Allin!
Now it is obvious that the slow speed of dbnomics 0.4 with or without
metadata is related to the fact that both are using same new API of
dbnomics.
Same problem with the failed statement of a single request as I just
figured out: The installed version of dbnomics 0.4 needs a correct
stated dataset. Monthly data like ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
runs only with a monthly dataset and not with a quaterly dataset. My
installed version of dbnomics 0.2 in gretl 2018c runs with both datasets
without an error.
Klaus
Am 29.03.2021 um 23:56 schrieb Allin Cottrell:
On Mon, 29 Mar 2021, klaus.hasenbach(a)web.de wrote:
> Thank you, Allin!
>
> But I guess there is a bug in dbnonmics 0.4 with Windows 10. Bundle
> requests via mask are slow as before (see for examble the attached
> scipt).
I see what you mean. Running a slightly cut-down version of your
multi-series request, with mask "DEU+GBR+IAA..Q" and a maximum of 250
series, I'm seeing timings and JSON sizes as follows:
dbnomics 0.4, with metadata: 0m56.569s, 2317360 bytes
dbnomics 0.4, no metadata: 0m55.649s, 2296071 bytes
So cutting out the metadata makes little difference to the time or
size. (Up till now we haven't been setting metadata=0 for this sort
of request -- we missed this case -- but it hardly helps.)
If I invoke the dbnomics v21 API instead, I get this:
time 0m4.886s, size 1905768 bytes
I still have to determine what's taking 10 times as long: dbnomics or
gretl's processing of the data. But at this point my suspicion is that
it's dbnomics.
> And single requests such as
>
> data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z are aborted immediatly with
> error:
> "Keine Datenbank wurde geöffnet"=no databank was opened
I'm not able to replicate that. I tried on Windows 10 today and the
series arrived immediately.
Allin
_______________________________________________
Gretl-users mailing list -- gretl-users(a)gretlml.univpm.it
To unsubscribe send an email to gretl-users-leave(a)gretlml.univpm.it
Website: https://gretlml.univpm.it/postorius/lists/gretl-users.gretlml.univpm.it/
3:07 p.m.
On Tue, 30 Mar 2021, klaus.hasenbach(a)web.de wrote:
Thank`s Allin!
Now it is obvious that the slow speed of dbnomics 0.4 with or without
metadata is related to the fact that both are using same new API of
dbnomics. [...]
Klaus, did you see my follow-up? The dbnomics API is not the primary
issue with the speed problem, which I think is now solved.
Same problem with the failed statement of a single request as I just
figured out: The installed version of dbnomics 0.4 needs a correct
stated dataset. Monthly data like ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
runs only with a monthly dataset and not with a quaterly dataset. My
installed version of dbnomics 0.2 in gretl 2018c runs with both datasets
without an error.
Not sure I'm understanding the problem. Below are three variants of
loading the series ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z, all working
here with gretl's dbnomics 0.41.
# (1) let the dbnomics import create a monthly dataset
clear
open dbnomics
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
smpl
# gives: 1993:01 - 2021:02 (n = 338)
# (2) start with a correctly sized monthly dataset
clear
nulldata 338
setobs 12 1993:01
open dbnomics
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
# start with a quarterly dataset (automatic compaction)
clear
nulldata 112
setobs 4 1993:1
open dbnomics
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
smpl
# gives: 1993:1 - 2020:4 (n = 112)
Allin
7:11 p.m.
No I did not see your follow-up til yet, Allin!
That`s realy great that you found a way to increase the speed of the
planned version of dbnomics 0.41. I will be very curious to run it!
Concerning the raised issue of data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
I reconized that I wrote the sequence of the opening comands totaly
wrong. See my comment below.
But funny it works fine with dbnomics 0.2 in gretl 2018c. ...Good for
someone like me who has almost no knowledge in computer programming anymore.
Klaus
Am 30.03.2021 um 17:07 schrieb Allin Cottrell:
On Tue, 30 Mar 2021, klaus.hasenbach(a)web.de wrote:
> Thank`s Allin!
>
> Now it is obvious that the slow speed of dbnomics 0.4 with or without
> metadata is related to the fact that both are using same new API of
> dbnomics. [...]
Klaus, did you see my follow-up? The dbnomics API is not the primary
issue with the speed problem, which I think is now solved.
> Same problem with the failed statement of a single request as I just
> figured out: The installed version of dbnomics 0.4 needs a correct
> stated dataset. Monthly data like ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
> runs only with a monthly dataset and not with a quaterly dataset. My
> installed version of dbnomics 0.2 in gretl 2018c runs with both datasets
> without an error.
Not sure I'm understanding the problem. Below are three variants of
loading the series ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z, all working
here with gretl's dbnomics 0.41.
# (1) let the dbnomics import create a monthly dataset
clear
open dbnomics
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
smpl
# gives: 1993:01 - 2021:02 (n = 338)
# (2) start with a correctly sized monthly dataset
clear
nulldata 338
setobs 12 1993:01
open dbnomics
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
start with a quarterly dataset (automatic compaction)
clear
nulldata 112
setobs 4 1993:1
open dbnomics
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
smpl
# gives: 1993:1 - 2020:4 (n = 112)
# Allin, put "open dbnomics" to the top and don`t use
"clear" then it
will produce the mentioned error in dbnomics 0.4 with gretl 2021a, not
so in 0.2 with 2018c (it is not because the wrong frequecy as I
thought before) !!!
open dbnomics
nulldata 112
setobs 4 1993.1
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
Allin
_______________________________________________
Gretl-users mailing list -- gretl-users(a)gretlml.univpm.it
To unsubscribe send an email to gretl-users-leave(a)gretlml.univpm.it
Website:
https://gretlml.univpm.it/postorius/lists/gretl-users.gretlml.univpm.it/
8:58 p.m.
On Tue, 30 Mar 2021, klaus.hasenbach(a)web.de wrote:
That's realy great that you found a way to increase the speed of
the planned version of dbnomics 0.41. I will be very curious to
run it!
I hope it works for you.
Concerning the raised issue of data
ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
I reconized that I wrote the sequence of the opening comands totaly
wrong. See my comment below.
The order of the commands doesn't matter so long as "open dbnomics"
comes before a "data ..." command. I used "clear" in my examples
only to emphasize the point, "I'm starting from scratch here". It's
not really necessary since "nulldata" clears the gretl workspace
by itself.
Here's your variant of the ECB series download:
open dbnomics
nulldata 112
setobs 4 1993.1
data ECB/IRS/M.IT.L.L40.CI.0000.EUR.N.Z
That too works fine for me with current gretl git/snapshot and
dbnomics 0.41. OK, that's on Linux, but I don't think there should
be any difference by platform.
However, it's possible that some changes since the 2021a release may
be helpful in this context. If you're able to test with a current
snapshot that would be great. If not, that's OK; I can rebuild 2021a
for testing.
Allin
1348
days inactive
1370
days old
11 comments
2 participants
participants (2)
-
Allin Cottrell -
klaus.hasenbach＠web.de