Inefficiency in join command?
by atecon
Hi all,
I just have to work a with a large panel dataset (left-hand side) to
which I would like to join a couple of series from a RHS-dataset. The
correct mapping is done via two keys.
I did some performance check, and it seems that the current
implementation runs the sorting/ mapping for each series joined
separately even though a single sorting/ mapping should be sufficient
(if I am not wrong).
In a first experiment I join all series from the RHS dataset by means of
the wildcard operator:
<join "@NAME_RHS_DATA" * --ikey=datedim,unitdim>
which takes about 5 sec. here.
Then I re-run the experiment by successively increasing the number of
series to join:
<hansl>
loop i=1..nelem(RHS_SERIES_NAMES)
printf "\nInfo: Start joining %d series.\n", $i
flush
strings tojoin = RHS_SERIES_NAMES[1:$i]
set stopwatch
join "@NAME_RHS_DATA" tojoin --ikey=datedim,unitdim
printf "\nInfo: Joining took %.2f sec.\n", $stopwatch
flush
list New = dataset - Base
delete New --force
endloop
</hansl>
The output is as follows:
<output>
Info: Joining all series took 4.91 sec.
Info: Start joining 1 series.
Info: Joining took 1.91 sec.
Info: Start joining 2 series.
Info: Joining took 2.88 sec.
Info: Start joining 3 series.
Info: Joining took 3.88 sec.
Info: Start joining 4 series.
Info: Joining took 4.84 sec.
Script done
</output>
Do you agree that the sorting or mapping overhead can in principle be
reduced when joining multiple series at once?
Thanks,
Artur
2 years, 10 months
string vs. strings (array) type issues (and bundles)
by Sven Schreiber
Hi, I'm tripping over subtleties of 1-element arrays, combined with bundles.
First, indexing into a strings array appears to return a string type
instead of a 1-element array. (For example: eval defarray("A","B")[2]
gives the string "B".) I guess that's intended, OK.
Secondly, however, if I want to stuff such a single string into a bundle
as an array, I get an error:
<hansl>
bundle b = null
strings b.S = "A" # error: expected strings, got string
</hansl>
Note that doing this without the bundle works fine, as expected:
<hansl>
strings aS = "A"
print aS # gives 1-element strings array
</hansl>
Haven't checked whether other arrays (matrices...) have the same problem.
thanks
sven
2 years, 11 months
shell escape on Windows: issue with quited path strings
by Sven Schreiber
Hi,
I'm seeing this phenomenon I don't quite understand when using a shell
call from within a gretl script (with "!"). Without going into the
details of what is actually called, the pattern of the following line
works alright, with string substitution:
! "C:\mypath\some.exe" C:\morepath\input.csv > C\morepath\output.txt
Notice that the exe path is already enclosed in quotes, although it's
not really needed; it also works without them in this setup without
embedded spaces. The following variants also work:
! C:\mypath\some.exe C:\morepath\input.csv > C\morepath\output.txt
! C:\mypath\some.exe "C:\morepath\input.csv" > "C\morepath\output.txt"
However, when I enclose everything in quotes (to be more portable, in
case the user name does contain a space) then it fails, i.e. the
expected output files are not created! Like this:
! "C:\mypath\some.exe" "C:\morepath\input.csv" > "C\morepath\output.txt"
I find this really puzzling - is something going wrong on the way from
gretl to the Windows shell, or am I simply missing something?
thanks
sven
3 years
not marked for translation
by Ignacio Diaz-Emparanza
Hi,
I have just seen a phrase that seems not to be marked for translation:
"If you meant to test for equality, please use '=='"
this appears in the GUI in "/sample/restrict from criterion" when you
use a unique '=' to define it.
--
Ignacio Díaz-Emparanza
Departamento de Métodos Cuantitativos
Universidad del País Vasco - Euskalherriko Unibertsitatea, UPV/EHU
Tfno: (+34) 94 601 3732
3 years, 1 month
join to dataset with string-values series
by atecon
Hi all,
I've re-compiled gretl this morning using the latest git master state. A
script which worked last Friday does not work anymore when trying to
join a numerical time-series to a dataset which comprises both numeric
and string-valued series. The error is "Cannot overwrite entire
string-valued series".
I vaguely remember that in some very recent conversation something on
string-valued series was mentioned. Could anybody refresh my mind on
this, please? ;-)
Best,
Artur
3 years, 1 month
foreach loop with on-the-fly array
by Sven Schreiber
Hi,
not sure whether it's a bug or simply a known limitation of the parser:
loop foreach i varnames(sth)
doesn't work, but
vn = varnames(sth)
loop foreach i vn
does.
I know that this array-loop thing is fairly new in any case.
This is with 2021b.
cheers
sven
3 years, 1 month
varlist inside function (and furthermore strange append effect)
by Sven Schreiber
Hi,
for the first thing, here's a minimal example:
<hansl>
function void checko(list L)
list L print # d_LRY d_LRM (expected result)
varlist # weird: 0) const 5) [masked] 6) [masked]
end function
open denmark
list L = LRY LRM
checko(diff(L))
</hansl>
OK, then in my use case I'm also using 'append' (inside the function) to
load data from a csv file. The first part of that csv file contains the
same data as in the list L, with the same series names. However,
afterwards it seems that for example d_LRY appears twice in the dataset
- at least it's the result of an empty "print". ("varlist" doesn't show
duplicates, but again has these [masked] entries as before with large
series ID# gaps.)
How to avoid the duplication and get a "sane" dataset view inside the
function?
thanks
sven
3 years, 1 month
checking the status of "set" options
by Sven Schreiber
Hi,
below is a hansl function that checks whether shell calls are enabled,
by scanning the output of "set" for the line " shell_ok = 1".
<hansl>
function scalar checkshell(void)
string setbuf
outfile --buffer=setbuf
set
end outfile
strings setparts = strsplit(setbuf, "\n")
return nelem(instrings(setparts, " shell_ok = 1")) # mind the
leading blank!
end function
</hansl>
This could be generalized, but the whole thing strikes me as relatively
complicated. I have the feeling we discussed this before, but I don't
remember any details. What's the best way to query the currently active
set options programmatically?
thanks
sven
3 years, 1 month
Status of extra 1.1
by Sven Schreiber
Hi,
is v1.1 of the "extra" addon still under development? Otherwise (if it
is finalized) I guess it should be made visible if one does Help /
Search for addons. (For me extra 1.0 is marked as current there.)
thanks
sven
3 years, 1 month
Gretl icon view enhancement
by Henrique Andrade
Dear gretl team,
What do you think about increasing the number of lines of the text
legend inside the gretl icon view?
Please take a look at the attached pdf file. It shows that besides the
space is sufficient for the English locale, it is not for some other
locales.
Best,
Henrique Andrade
3 years, 1 month
