10:32 a.m.
Hi gretl-listers,
I have a problem with running times of a simulation of mine. IMO the
time needed should grow linearly with the number of simulation runs, but
it turns out that instead it is the "marginal cost" of an extra 100 runs
which grows linearly; each additional 100 runs seem to take about 4 or 5
seconds more than the previously added 100 runs (see below for the numbers).
This essentially simulates data and estimates Vecms. For 200 runs, data
simulation takes about 1/3 of the total time (5s), the estimation close
to the rest (9s, 1s is remaining overhead). The sorting of the results
with gretl's quantile() function is not significantly costly. I don't
think that RAM limitations are a problem, it's not that much data
actually, it's just CPU intensive.
One guess is that the huge script text output may be responsible for
part of the problem -- here it would help if a VECM could be estimated
without any output, as I was suggesting in a previous email (vecm with
the --silent option).
But that is just one guess, and I don't really know what's happening.
Any hints or remarks? I'm not posting the code because it's quite complex.
thanks,
sven
100: Elapsed time: 5.530000
(Diff: 10)
200: Elapsed time: 15.090000
(Diff: 14)
300: Elapsed time: 29.180000
(Diff: 19)
400: Elapsed time: 47.700000
(Diff: 23)
500: Elapsed time: 70.750000
(Diff: 27)
600: Elapsed time: 98.170000
(Diff: 32)
700: Elapsed time: 130.090000
(Diff: 39 -- gretl not always the only active task)
800: Elapsed time: 168.830000 (35810 lines in script output, 1.6MB)
(Diff: 41)
900: Elapsed time: 209.550000
(Diff: 47)
1000: Elapsed time: 257.090000 (51110 lines in script output, 2.3MB)
20000: ran overnight, no result, gretl still at 100% CPU, killed it.
11:38 a.m.
On Sun, 10 Feb 2008, Sven Schreiber wrote:
Hi gretl-listers,
I have a problem with running times of a simulation of mine.
Hmm. Interesting. Could you please post a simplified version of the
script that exposes the problem?
One guess is that the huge script text output may be responsible for
part of
the problem -- here it would help if a VECM could be estimated without any
output, as I was suggesting in a previous email (vecm with the --silent
option).
On linux, you have a poor man's version of the --quiet switch, that is
redirecting output to /dev/null, as in
outfile "/dev/null" --write
vecm 2 3 1 2 3
outfile "/dev/null" --close
I don't know if anything similar exists on Windows.
Riccardo (Jack) Lucchetti
Dipartimento di Economia
Università Politecnica delle Marche
r.lucchetti(a)univpm.it
http://www.econ.univpm.it/lucchetti
2:42 p.m.
Am 10.02.2008 12:38, Riccardo (Jack) Lucchetti schrieb:
On Sun, 10 Feb 2008, Sven Schreiber wrote:
> One guess is that the huge script text output may be responsible
for
> part of the problem -- here it would help if a VECM could be estimated
> without any output, as I was suggesting in a previous email (vecm with
> the --silent option).
On linux, you have a poor man's version of the --quiet switch, that is
redirecting output to /dev/null, as in
outfile "/dev/null" --write
vecm 2 3 1 2 3
outfile "/dev/null" --close
I don't know if anything similar exists on Windows.
Well luck has it that I'm currently working on Linux again, and your
idea was brilliant -- the time for 500 runs is down from 71 to 17, and
for 1000 runs from 257 to 35, so now it's linear as expected! Thanks a
lot for solving this problem!
(BTW, I had to put the 'outfile "/dev/null" --write' command right into
the function where the 'vecm' command is; it wouldn't work to wrap the
top-level function call in it. It smells like a little bug to me, but
I'm not sure.)
three cheers for Jack :-)
sven
1:22 a.m.
On Sun, 10 Feb 2008, Riccardo (Jack) Lucchetti wrote:
On linux, you have a poor man's version of the --quiet switch,
that is
redirecting output to /dev/null, as in
outfile "/dev/null" --write
vecm 2 3 1 2 3
outfile "/dev/null" --close
A couple of observations here.
First, I've added a portable version of the above:
outfile null --write
...
outfile --close
Using the keyword "null" in place of a real filename disables
gretl's printing functions -- they become no-ops.
Second, Sven, if you're using plain "gretl" (i.e. gretl_x11) for
large-scale simulations, don't. Use gretlcli instead. This saves
you all the GUI overhead.
Allin.
9:38 a.m.
Am 11.02.2008 02:22, Allin Cottrell schrieb:
On Sun, 10 Feb 2008, Riccardo (Jack) Lucchetti wrote:
> On linux, you have a poor man's version of the --quiet switch, that is
> redirecting output to /dev/null, as in
>
> outfile "/dev/null" --write
> vecm 2 3 1 2 3
> outfile "/dev/null" --close
A couple of observations here.
First, I've added a portable version of the above:
outfile null --write
...
outfile --close
Using the keyword "null" in place of a real filename disables
gretl's printing functions -- they become no-ops.
very nice!
Second, Sven, if you're using plain "gretl" (i.e. gretl_x11) for
large-scale simulations, don't. Use gretlcli instead. This saves
you all the GUI overhead.
Good point -- although with the /dev/null trick it's running ok for me.
But I'm not sure if what I do already qualifies as "large-scale".
Anyway, I will experiment a little and see if using gretcli provides
another noticeable speedup.
thanks,
sven
10:18 a.m.
Am 11.02.2008 10:38, Sven Schreiber schrieb:
Am 11.02.2008 02:22, Allin Cottrell schrieb:
> On Sun, 10 Feb 2008, Riccardo (Jack) Lucchetti wrote:
>
>> On linux, you have a poor man's version of the --quiet switch, that is
>> redirecting output to /dev/null, as in
>>
>> outfile "/dev/null" --write
>> vecm 2 3 1 2 3
>> outfile "/dev/null" --close
>
> A couple of observations here.
>
> First, I've added a portable version of the above:
>
> outfile null --write
> ...
> outfile --close
>
> Using the keyword "null" in place of a real filename disables gretl's
> printing functions -- they become no-ops.
very nice!
>
However, I do get a file called "null" in the working dir with the
estimation results, so this does not yet seem to work properly... (fresh
cvs)
thanks,
sven
3:05 p.m.
On Mon, 11 Feb 2008, Sven Schreiber wrote:
> > outfile null --write
> > ...
> > outfile --close
> >
> > Using the keyword "null" in place of a real filename disables
gretl's
> > printing functions -- they become no-ops.
However, I do get a file called "null" in the working dir with
the estimation results, so this does not yet seem to work
properly... (fresh cvs)
Hmm. What do you see in response to the initial outfile command?
Given the script
open data4-10
outfile null --write
print 1 2 3 4
outfile --close
I'm getting:
? outfile null --write
Now discarding output
Closed output file 'null'
Done
The key is "Now discarding output": if you see that, no file has
been opened.
Allin.
3:14 p.m.
Am 11.02.2008 16:05, Allin Cottrell schrieb:
On Mon, 11 Feb 2008, Sven Schreiber wrote:
>>> outfile null --write
>>> ...
>>> outfile --close
>>>
>>> Using the keyword "null" in place of a real filename disables
gretl's
>>> printing functions -- they become no-ops.
> However, I do get a file called "null" in the working dir with
> the estimation results, so this does not yet seem to work
> properly... (fresh cvs)
Hmm. What do you see in response to the initial outfile command?
Given the script
open data4-10
outfile null --write
print 1 2 3 4
outfile --close
I'm getting:
? outfile null --write
Now discarding output
Closed output file 'null'
Done
The key is "Now discarding output": if you see that, no file has
been opened.
I get sth different (translated from my German version):
Now writing output to '/home/<workdir setting>/null'
Closed output file '/home/<workdir setting>/null'
fyi (if it's relevant), my workdir settings are to *not* use the startup
dir from the shell, but the pre-configured workdir.
-sven
4:48 p.m.
On Mon, 11 Feb 2008, Sven Schreiber wrote:
> I'm getting:
>
> ? outfile null --write
> Now discarding output
> Closed output file 'null'
> Done
>
> The key is "Now discarding output": if you see that, no file has been
> opened.
I get sth different (translated from my German version):
Now writing output to '/home/<workdir setting>/null'
Closed output file '/home/<workdir setting>/null'
fyi (if it's relevant), my workdir settings are to *not* use the
startup dir from the shell, but the pre-configured workdir.
You're right, that did make a difference. Fixed in CVS.
Allin.
9:58 a.m.
New subject: little bug with dire consequences (winsnapshot)
Dear listers, greetings from Nogent-sur-Marne (~ Paris) !
I noticed yesterday a seemingly innocuous bug that crashes gretl.
Editing my scripts, I mistakenly typed :
matrix W = {x~y~z}
where x, y and z are series name. I excuted the code on just that line:
GRETL crashes.
It seems that it should say something like " , or ; expected but ~ found
instead" rather than crash.
It took me some time to figure that out because initially I throught the
problem I reported lately was poping up again. Fortunately, this is not
the case, but this minor bug has real serious consequences so that I
guess it is worth reporting it.
The bug is affecting the windows snapshot i downloaded last week (friday
8-2-08).
Cheers
Franck
10:04 a.m.
New subject: little bug with dire consequences (winsnapshot)
Am 12.02.2008 10:58, nadaud(a)centre-cired.fr schrieb:
Dear listers, greetings from Nogent-sur-Marne (~ Paris) !
I noticed yesterday a seemingly innocuous bug that crashes gretl.
Editing my scripts, I mistakenly typed :
matrix W = {x~y~z}
where x, y and z are series name. I excuted the code on just that line:
GRETL crashes.
It seems that it should say something like " , or ; expected but ~ found
instead" rather than crash.
It took me some time to figure that out because initially I throught the
problem I reported lately was poping up again. Fortunately, this is not
the case, but this minor bug has real serious consequences so that I
guess it is worth reporting it.
The bug is affecting the windows snapshot i downloaded last week (friday
8-2-08).
On yesterday's cvs I get an informative error message, so I think you
became a victim of the temporary genr breakage (devel version only) that
Allin has fixed in the meantime. A newer win snapshot should solve this
(don't know if it's already available).
cheers,
sven
2:50 p.m.
New subject: little bug with dire consequences (winsnapshot)
On Tue, 12 Feb 2008, Sven Schreiber wrote:
Am 12.02.2008 10:58, nadaud(a)centre-cired.fr schrieb:
> Dear listers, greetings from Nogent-sur-Marne (~ Paris) !
>
> I noticed yesterday a seemingly innocuous bug that crashes gretl.
>
> Editing my scripts, I mistakenly typed :
>
> matrix W = {x~y~z} ...
On yesterday's cvs I get an informative error message, so I
think you became a victim of the temporary genr breakage (devel
version only) that Allin has fixed in the meantime. A newer win
snapshot should solve this (don't know if it's already
available).
Yes, the Windows snapshot is up to date with recent genr fixes.
Allin.
11:56 a.m.
Am 11.02.2008 10:38, Sven Schreiber schrieb:
Am 11.02.2008 02:22, Allin Cottrell schrieb:
>
> Second, Sven, if you're using plain "gretl" (i.e. gretl_x11) for
> large-scale simulations, don't. Use gretlcli instead. This saves you
> all the GUI overhead.
Good point -- although with the /dev/null trick it's running ok for me.
But I'm not sure if what I do already qualifies as "large-scale".
Anyway, I will experiment a little and see if using gretcli provides
another noticeable speedup.
Using
gretlcli -b myscript.inp > clilog.txt
I don't save any time with respect to running the script inside GUI
gretl and applying the 'outfile null --write' trick.
But I notice what may be a slight bug: myscript.inp is correctly read
from my configured workdir, but clilog.txt is saved in ~ (= $PWD I
guess) instead of the workdir. Or is this intended?
-sven
6:26 p.m.
On Tue, 12 Feb 2008, Sven Schreiber wrote:
Using
gretlcli -b myscript.inp > clilog.txt
I don't save any time with respect to running the script inside
GUI gretl and applying the 'outfile null --write' trick.
OK.
But I notice what may be a slight bug: myscript.inp is correctly
read from my configured workdir, but clilog.txt is saved in ~ (=
$PWD I guess) instead of the workdir. Or is this intended?
In the invocation above, redirection is done by the shell;
gretlcli has no idea where the output is going.
Allin.
6182
days inactive
6184
days old
13 comments
4 participants
participants (4)
-
Allin Cottrell -
nadaud＠centre-cired.fr -
Riccardo (Jack) Lucchetti -
Sven Schreiber