Inefficiency in join command?
by atecon
Hi all,
I just have to work a with a large panel dataset (left-hand side) to
which I would like to join a couple of series from a RHS-dataset. The
correct mapping is done via two keys.
I did some performance check, and it seems that the current
implementation runs the sorting/ mapping for each series joined
separately even though a single sorting/ mapping should be sufficient
(if I am not wrong).
In a first experiment I join all series from the RHS dataset by means of
the wildcard operator:
<join "@NAME_RHS_DATA" * --ikey=datedim,unitdim>
which takes about 5 sec. here.
Then I re-run the experiment by successively increasing the number of
series to join:
<hansl>
loop i=1..nelem(RHS_SERIES_NAMES)
printf "\nInfo: Start joining %d series.\n", $i
flush
strings tojoin = RHS_SERIES_NAMES[1:$i]
set stopwatch
join "@NAME_RHS_DATA" tojoin --ikey=datedim,unitdim
printf "\nInfo: Joining took %.2f sec.\n", $stopwatch
flush
list New = dataset - Base
delete New --force
endloop
</hansl>
The output is as follows:
<output>
Info: Joining all series took 4.91 sec.
Info: Start joining 1 series.
Info: Joining took 1.91 sec.
Info: Start joining 2 series.
Info: Joining took 2.88 sec.
Info: Start joining 3 series.
Info: Joining took 3.88 sec.
Info: Start joining 4 series.
Info: Joining took 4.84 sec.
Script done
</output>
Do you agree that the sorting or mapping overhead can in principle be
reduced when joining multiple series at once?
Thanks,
Artur
3 years, 1 month
gretl 2021c released
by Allin Cottrell
Apologies for cross-posting, but here's the change log for the 2021c
release, now available at http://gretl.sourceforge.net/
2021-08-30 version 2021c
- New function contains(): indicates whether a numerical
object contains members of a specified set
- New function bcheck(): aids in handling bundle arguments
to user-defined functions
- New command "panspec": substantially, the old "hausman"
command under a different name; for the moment, "hausman"
is still kept as an alias for backward compatibility
- fcstats() matrix row labels: distinguish U1 vs U2 variant
of Theil's U statistic
- msplitby(): allow for more flexible syntax
- printing of bundles: sort members by type and key
- "print" command: make more flexible, it now includes
most of the functionality of "eval"
- "var" command with --lagselect option: add a --minlag
option to allow testing against 0 lags, or imposing the
assumption that at least p > 1 lags are needed
- "set" command: new variable "datacols" to control the max
number of series to print side by side
- "foreign" command mechanism: updates to support version
4.1.0 of R and version 8.0 of Ox Console
- "eval" command: introduce "=" as an alias
- extend the ".=" operator to the "strings .= string" case
- GUI: introduce experimental features: plot collections and
integrated console
- GUI: add option to use the tab key as the trigger for
autocompletion in the script editor and console
- GUI: add command-line options: --new to start a new gretl
instance unconditionally and --single to use a single
program instance unconditionally
- GUI script editor: make the Find box appear at the foot of
the window, not in a separate dialog
- GUI: provide larger (24 pixel) toolbar icons for use on
small, high-density displays in particular
- GUI: numerous other small fixes and enhancements
- MS Windows, 64-bit: update to gnuplot 5.4.1, patched to
avoid too-big fonts on Windows with Ultra HD display
- Fix bug: spurious results from "system" command in case of
exact collinearity
- Fix bug: spurious reduction of the sample range in the grid
variant of "panplot" with some patterns of missing values
- Fix bug: wrong results from the polyfit() function when
applied to panel data
- Numerous additional small bug fixes
- Internals: make "libset" code more compact and efficient
- Internals: clear out old gettext variants that encoded
translations in a Windows code page rather than UTF-8
- gretl_bundle.h: rename argument named "template" to avoid
trouble with C++
- Substantial enhancements to the SVAR and regls addons
- New "logging" addon
- New, substantially updated Russian translation
- Cleanup: remove the "arbond" command, in favor of "dpanel",
and add some dpanel enhancements
--
Allin Cottrell
Department of Economics
Wake Forest University, NC
3 years, 1 month
a small generalization of the replace() interface
by Sven Schreiber
Hi,
the replace() function takes as 2nd and 3rd arguments the mapping pairs
of values. What about enabling a little bit of syntactic sugar for the
case of a matrix with exactly 2 columns, standing in for the respective
vectors, making the 3rd argument optional in cases like the following
example:
matrix m = {3, 0.5; 2, 0.7} # arbitrary stuff for the example
series y = replace(x, m) # not working yet
which would be equivalent to:
series y = replace(x, m[,1], m[,2])
BTW, thinking about this, there seems to be some similarity to the
strsub() function. What I mean is that replace() could be overloaded to
nest the usage of strsub(), based on the type of the first argument. And
strings arrays might be supported as arguments as well. This is just a
general observation, I don't have a concrete need right now, but I guess
there would be use cases, replacing some handwritten loops.
Does replace() actually work already on a string-valued series? If not,
I guess it should?
thanks
sven
3 years, 1 month
Static code analysis of gretl
by Hélio Guilherme
Hello,
I finally got to setup a SonarCloud analysis of Gretl.
I have created a GitHub clone of the SourceForge original code, and added a
configuration for SonarCloud.
You may give a look to the analysis, but please don't lose focus on the
delivery of the new version of Gretl.
Maybe there are some insights that could help improving the code.
https://sonarcloud.io/dashboard?id=HelioGuilherme66_gretl
Cheers,
Hélio
3 years, 1 month
function package window on Ubuntu: one invisible icon
by Sven Schreiber
Hi,
looking at a current git build on Ubuntu with the Budgie desktop I'm
"missing" the icon "look on server" in the window with the function
packages list. But the button is there and becomes visible on mouse-over.
Don't know whether it's a new phenomenon, because I'm not running gretl
there very often.
Might very well be specific to Budgie and whatever it's using.
cheers
sven
3 years, 1 month
translation, possible language typo
by Ioannis A. Venetis
Trying to coplete full translation I found the following in el.po (lines
4344-4346).
There is a language typo in 4346
#: lib/src/gretl_utils.c:1192 lib/src/gretl_utils.c:1217
#, c-format
msgid "frequency (%d) does not make seem to make sense"
Ioannis A. Venetis
3 years, 1 month
auto-indentation: tiny remaining issue
by Sven Schreiber
Hi,
one more thing which may be nice to fix for the release:
I'm seeing (on Win and Linux) the following problem in placing the first
line of code like this (hope the spacing goes through the email):
function void F(int a,
int b)
matrix x
(The line starts 2 columns behind the start of the function argument
signature. I'm expecting "matrix" to show up with one indentation level
from the start, so with 4 blanks for me.)
Necessary ingredients are having the arguments on multiple lines, and
having a whitespace separating line. The problem is that I'm using
exactly those ingredients a lot, and I think they're natural.
I can imagine it's not easy to get this additional requirement defined
in a regex or whatever is used. I could perhaps give it a try if you
show me where it is.
thanks
sven
3 years, 1 month
string substitution with bundle elements
by Sven Schreiber
Hi,
we may have discussed this before, not sure. I guess the following is
simply not supported and not a bug?
<hansl>
open denmark
string a = "quiet"
bundle b = defbundle("a","quiet")
ols LRM 0 --@a # works
ols LRM 0 --(a)b.a # fails
</hansl>
thanks
sven
3 years, 1 month
The 2021c release candidate is available (with apologies for cross-posting)
by Riccardo (Jack) Lucchetti
Hi all,
on behalf of the development team, I'm glad to announce that we're
about to release gretl 2021c. The official release will take place in
a week or so, but you can already see what's new by downloading and
installing the 2021c release candidate:
64-bit Windows:
https://sourceforge.net/projects/gretl/files/snapshots/gretl_install-64.e...
Mac OS (Intel):
https://sourceforge.net/projects/gretl/files/snapshots/gretl-macos-intel....
Mac OS (M1):
https://sourceforge.net/projects/gretl/files/snapshots/gretl-macos-arm64....
To have a quick idea of what the new stuff is, take look at the
Changelog:
http://gretl.sourceforge.net/ChangeLog.html
In fact, we'd appreciate it very much if you could give these versions a
good go: we have tried to squash all the bugs we were aware of, but you
can never be 100% sure, so please hammer on the release candidates as much
as you can. Some of the new GUI features are somewhat experimental, so
we'd be grateful if you could try them out. To enable them, go to
Tools>Preferences>General and look for the following:
1) the "plot collection" feature (multiple plots in one window); see
the help for "set plot_collection"
2) the "Main window includes console" feature (try it for yourselves,
I don't want to spoil it for you).
Between now and the official release, we'll only fix any bugs you may
report (no new features), and give the translators some time to send us
updates of the PO files. As of now, the only localisations that are 100%
complete are Italian and Russian (thanks to Olesya Petrenko).
Should you have any question or comment, please feel free to contact
any of us.
Thanks!
PS: Linux people will have to download the source and compile it
themselves; the source code is at
https://sourceforge.net/projects/gretl/files/snapshots/gretl-source.tar.x...
If you need guidance on how to build, have a look at Appendix B in the
User's Guide, or watch the excellent video by Clive Nicholas:
https://youtu.be/WSdZkd-uln4
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
3 years, 1 month
