12:17 p.m.
Hi all,
I just have to work a with a large panel dataset (left-hand side) to
which I would like to join a couple of series from a RHS-dataset. The
correct mapping is done via two keys.
I did some performance check, and it seems that the current
implementation runs the sorting/ mapping for each series joined
separately even though a single sorting/ mapping should be sufficient
(if I am not wrong).
In a first experiment I join all series from the RHS dataset by means of
the wildcard operator:
<join "@NAME_RHS_DATA" * --ikey=datedim,unitdim>
which takes about 5 sec. here.
Then I re-run the experiment by successively increasing the number of
series to join:
<hansl>
loop i=1..nelem(RHS_SERIES_NAMES)
printf "\nInfo: Start joining %d series.\n", $i
flush
strings tojoin = RHS_SERIES_NAMES[1:$i]
set stopwatch
join "@NAME_RHS_DATA" tojoin --ikey=datedim,unitdim
printf "\nInfo: Joining took %.2f sec.\n", $stopwatch
flush
list New = dataset - Base
delete New --force
endloop
</hansl>
The output is as follows:
<output>
Info: Joining all series took 4.91 sec.
Info: Start joining 1 series.
Info: Joining took 1.91 sec.
Info: Start joining 2 series.
Info: Joining took 2.88 sec.
Info: Start joining 3 series.
Info: Joining took 3.88 sec.
Info: Start joining 4 series.
Info: Joining took 4.84 sec.
Script done
</output>
Do you agree that the sorting or mapping overhead can in principle be
reduced when joining multiple series at once?
Thanks,
Artur
10:26 a.m.
Am 31.03.2021 um 14:17 schrieb atecon:
In a first experiment I join all series from the RHS dataset by means
of
the wildcard operator:
<join "@NAME_RHS_DATA" * --ikey=datedim,unitdim>
which takes about 5 sec. here.
Then I re-run the experiment by successively increasing the number of
series to join:
<hansl>
loop i=1..nelem(RHS_SERIES_NAMES)
printf "\nInfo: Start joining %d series.\n", $i
flush
strings tojoin = RHS_SERIES_NAMES[1:$i]
set stopwatch
join "@NAME_RHS_DATA" tojoin --ikey=datedim,unitdim
printf "\nInfo: Joining took %.2f sec.\n", $stopwatch
flush
list New = dataset - Base
delete New --force
endloop
</hansl>
As some of you may have noticed, since back in March a feature request
ticket had been created and some discussion took place there
(https://sourceforge.net/p/gretl/feature-requests/151/), although it's
perhaps debatable whether it was instead a mild bug, but anyway.
I believe in current git Allin has worked on fixing this, but I haven't
tested myself yet - part of the reason was that Artur's script above was
not self-contained.
So, Artur, could you test your setup on current git, or alternatively
post a self-contained example for us to test?
thanks
sven
2:53 p.m.
Am 10.09.21 um 12:26 schrieb Sven Schreiber:
Am 31.03.2021 um 14:17 schrieb atecon:
> In a first experiment I join all series from the RHS dataset by means of
> the wildcard operator:
> <join "@NAME_RHS_DATA" * --ikey=datedim,unitdim>
> which takes about 5 sec. here.
>
> Then I re-run the experiment by successively increasing the number of
> series to join:
>
> <hansl>
> loop i=1..nelem(RHS_SERIES_NAMES)
> printf "\nInfo: Start joining %d series.\n", $i
> flush
> strings tojoin = RHS_SERIES_NAMES[1:$i]
>
> set stopwatch
> join "@NAME_RHS_DATA" tojoin --ikey=datedim,unitdim
>
> printf "\nInfo: Joining took %.2f sec.\n", $stopwatch
> flush
>
> list New = dataset - Base
> delete New --force
> endloop
> </hansl>
As some of you may have noticed, since back in March a feature request
ticket had been created and some discussion took place there
(https://sourceforge.net/p/gretl/feature-requests/151/), although it's
perhaps debatable whether it was instead a mild bug, but anyway.
I believe in current git Allin has worked on fixing this, but I haven't
tested myself yet - part of the reason was that Artur's script above was
not self-contained.
So, Artur, could you test your setup on current git, or alternatively
post a self-contained example for us to test?
Hi,
This is great news. I'll try to test this over the weekend. Thanks for
your effort, Allin and Sven!
Artur
6:52 p.m.
On Fri, 10 Sep 2021, Sven Schreiber wrote:
[shows incomplete example from Artur, suggesting inefficiency in
"join"]
As some of you may have noticed, since back in March a feature
request
ticket had been created and some discussion took place there
(https://sourceforge.net/p/gretl/feature-requests/151/)
I believe in current git Allin has worked on fixing this, but I haven't
tested myself yet - part of the reason was that Artur's script above was
not self-contained.
Looking into the join code it became apparent that there was at
least one possible source of inefficiency. That is, if you're
importing n series via a single invocation of the join command, we
were calculating the matching of inner and outer keys n times, where
in principle we could do this once, store the results and apply them
to each of the imports -- since when you specify multiple series for
joining, the keys (if any) must be the same for all of them. But
note that this is not an entirely "free lunch", since storing the
matching results requires allocation of extra memory that's not
needed otherwise.
In current git we employ the "calculate once and store" method for
the first key (not yet for the second, if present, but most of the
work goes on the first one).
I've tested an example where the dataset has 20000 observations and
there are 20 series to import in one go, with a single matching key
and aggregation via the average of the matching values. What I found
was a speed-up of about 2 or 3 percent with the new method. So with
these parameters it appears that the key-matching code actually
takes a trivial proportion of the overall compute time, hardly worth
bothering with.
At this point it would be good to have an example which exhibits a
substantial (supra-linear) slowdown on importing more series. Maybe
that's the case with Artur's example. Anyway, a minimal but
informative test case would be very useful.
Allin
1732
days inactive
1895
days old
4 comments
4 participants
participants (4)
-
Allin Cottrell -
Artur T. -
atecon
-
Sven Schreiber