12:36 p.m.
Hi,
I'm trying out something that involves launching external programs, and
I have (re-) discovered the "launch" command (only actively remembered
the "!" command for synchronous invocation).
But I noticed a surprising difference which I think could be a bug:
While gretl's current CWD setting is passed to the external program as
its active directory when using "!", another path/directory is used when
using "launch".
That other directory is not gretl's dotdir, either. My suspicion is that
it might be the last-opened directory (the one which is also remembered
by the datafile-open window, if the user chooses it that way).
I hope that explanation was clear enough. I don't see any reason why "!"
and "launch" should behave differently in that respect.
(Also, "launch" isn't syntax-colored.)
thanks,
sven
2:12 p.m.
On Fri, 6 Jan 2017, Sven Schreiber wrote:
Hi,
I'm trying out something that involves launching external programs, and I
have (re-) discovered the "launch" command (only actively remembered the
"!"
command for synchronous invocation).
But I noticed a surprising difference which I think could be a bug:
While gretl's current CWD setting is passed to the external program as its
active directory when using "!", another path/directory is used when using
"launch".
That other directory is not gretl's dotdir, either. My suspicion is that it
might be the last-opened directory (the one which is also remembered by the
datafile-open window, if the user chooses it that way).
I hope that explanation was clear enough. I don't see any reason why "!"
and
"launch" should behave differently in that respect.
For "launch" we were using a GLib convenience function that didn't
support setting a working directory for the child process. I've now
modified this to use a more complex function that allows for such a
setting.
Allin
5:12 p.m.
Am 06.01.2017 um 20:12 schrieb Allin Cottrell:
On Fri, 6 Jan 2017, Sven Schreiber wrote:
For "launch" we were using a GLib convenience function that
didn't
support setting a working directory for the child process. I've now
modified this to use a more complex function that allows for such a
setting.
That's great, thanks. I hope it also cures another side effect,
something that currently doesn't work with "launch":
When I do "! gretlcli.exe -b testscript.inp > testout.out",
(that's on Windows obviously) I get the expected testout.out file. But
when I instead use "launch", the output file doesn't seem to get created
anywhere.
I'll test with the next snapshot.
cheers,
sven
5:17 p.m.
On Fri, 6 Jan 2017, Sven Schreiber wrote:
Am 06.01.2017 um 20:12 schrieb Allin Cottrell:
> On Fri, 6 Jan 2017, Sven Schreiber wrote:
> For "launch" we were using a GLib convenience function that didn't
> support setting a working directory for the child process. I've now
> modified this to use a more complex function that allows for such a
> setting.
That's great, thanks. I hope it also cures another side effect, something
that currently doesn't work with "launch":
When I do "! gretlcli.exe -b testscript.inp > testout.out",
(that's on Windows obviously) I get the expected testout.out file. But when I
instead use "launch", the output file doesn't seem to get created
anywhere.
I'll test with the next snapshot.
OK. I don't know if redirection is going to work with the Windows
CreateProcess API -- we'll see! (When you use "!" we go via cmd.exe,
which does support redirection.)
Allin
5:23 p.m.
On Fri, 6 Jan 2017, Sven Schreiber wrote:
When I do "! gretlcli.exe -b testscript.inp >
testout.out",
(that's on Windows obviously) I get the expected testout.out file. But when I
instead use "launch", the output file doesn't seem to get created
anywhere.
I'll test with the next snapshot.
Of course your request makes sense, but just out of curiosity: why would
one run gretlcli from inside gretl?
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
4:54 a.m.
Am 06.01.2017 um 23:23 schrieb Riccardo (Jack) Lucchetti:
On Fri, 6 Jan 2017, Sven Schreiber wrote:
> When I do "! gretlcli.exe -b testscript.inp > testout.out",
> (that's on Windows obviously) I get the expected testout.out file. But
> when I instead use "launch", the output file doesn't seem to get
> created anywhere.
> I'll test with the next snapshot.
Of course your request makes sense, but just out of curiosity: why would
one run gretlcli from inside gretl?
I had an idea last night for a little function that could make it a
little easier to run the same script in parallel with different
parameter values. That's why I need "launch" instead of "!",
because as
per the doc "!" would mean that the main gretl process would wait each
time, destroying the parallelism.
So this is an "embarassingly trivial" type of parallelism, where each
worker gets its own gretlcli instance/process.
cheers,
sven
5:13 a.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
Am 06.01.2017 um 23:23 schrieb Riccardo (Jack) Lucchetti:
> On Fri, 6 Jan 2017, Sven Schreiber wrote:
>
>> When I do "! gretlcli.exe -b testscript.inp > testout.out",
>> (that's on Windows obviously) I get the expected testout.out file. But
>> when I instead use "launch", the output file doesn't seem to get
>> created anywhere.
>> I'll test with the next snapshot.
>
> Of course your request makes sense, but just out of curiosity: why would
> one run gretlcli from inside gretl?
>
I had an idea last night for a little function that could make it a little
easier to run the same script in parallel with different parameter values.
That's why I need "launch" instead of "!", because as per the doc
"!" would
mean that the main gretl process would wait each time, destroying the
parallelism.
So this is an "embarassingly trivial" type of parallelism, where each worker
gets its own gretlcli instance/process.
Sorry if I ask, but are you aware of the getenv()/ngetenv() functions?
Actually, we also have the "scriptopt" trick, although I don't think
it's
documented anywhere. As in:
<shell>
jack@asus-nb:~$ gretlcli -q --scriptopt=2.7181718
gretl version 2017a-git
"help" gives a list of commands
Type "open filename" to open a data set
? eval log(scriptopt)
0.99995952
?
</shell>
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
6:08 a.m.
Am 07.01.2017 um 11:13 schrieb Riccardo (Jack) Lucchetti:
On Sat, 7 Jan 2017, Sven Schreiber wrote:
> So this is an "embarassingly trivial" type of
parallelism, where each
> worker gets its own gretlcli instance/process.
Sorry if I ask, but are you aware of the getenv()/ngetenv() functions?
Actually, we also have the "scriptopt" trick, although I don't think
it's documented anywhere. As in:
<shell>
jack@asus-nb:~$ gretlcli -q --scriptopt=2.7181718
If that's just for a single scalar it's not enough I'm afraid --
although I guess one could simply use it as an index type value.
About getenv(): I had looked at it, but AFAICS it would only give me the
home directory, not the gretl CWD.
thanks,
sven
7:29 a.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
Am 07.01.2017 um 11:13 schrieb Riccardo (Jack) Lucchetti:
> On Sat, 7 Jan 2017, Sven Schreiber wrote:
>> So this is an "embarassingly trivial" type of parallelism, where each
>> worker gets its own gretlcli instance/process.
>
> Sorry if I ask, but are you aware of the getenv()/ngetenv() functions?
> Actually, we also have the "scriptopt" trick, although I don't think
> it's documented anywhere. As in:
>
> <shell>
> jack@asus-nb:~$ gretlcli -q --scriptopt=2.7181718
If that's just for a single scalar it's not enough I'm afraid -- although I
guess one could simply use it as an index type value.
About getenv(): I had looked at it, but AFAICS it would only give me the home
directory, not the gretl CWD.
Sorry, I wasn't clear. I meant something like this:
<shell>
jack@asus-nb:~$ X="{1,2,3}" a=11 gretlcli -q
gretl version 2017a-git
Disabling OpenBLAS multi-threading (OpenMP/pthreads collision)
"help" gives a list of commands
Type "open filename" to open a data set
? a = ngetenv("a")
Generated scalar a = 11
? sx = getenv("X")
Generated string sx
? X = @sx
Generated matrix X
</shell>
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
9:52 a.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
Am 06.01.2017 um 23:23 schrieb Riccardo (Jack) Lucchetti:
> On Fri, 6 Jan 2017, Sven Schreiber wrote:
>
>> When I do "! gretlcli.exe -b testscript.inp > testout.out",
>> (that's on Windows obviously) I get the expected testout.out file. But
>> when I instead use "launch", the output file doesn't seem to get
>> created anywhere.
>> I'll test with the next snapshot.
>
> Of course your request makes sense, but just out of curiosity: why would
> one run gretlcli from inside gretl?
>
I had an idea last night for a little function that could make it a little
easier to run the same script in parallel with different parameter values.
That's why I need "launch" instead of "!", because as per the doc
"!" would
mean that the main gretl process would wait each time, destroying the
parallelism.
So this is an "embarassingly trivial" type of parallelism, where each worker
gets its own gretlcli instance/process.
"launch" is really intended for opening GUI applications from within
gretl.
It sounds as if MPI can do what you want here. Something like the
following:
<hansl>
# common setup, if applicable
scalar T = 100
scalar k = 3
matrix X
# set variant parameters here
if $mpirank == 0
X = mnormal(T, k)
elif $mpirank == 1
X = muniform(T, k)
endif
# do some work
matrix XXi = inv(X'X)
# write output
string outname = sprintf("output%d.txt", $mpirank)
outfile @outname --write
print XXi
outfile --close
</hansl>
<shell>
mpiexec -n 2 gretlmpi parallel.inp
</shell>
Allin
10:39 a.m.
Am 07.01.2017 um 15:52 schrieb Allin Cottrell:
On Sat, 7 Jan 2017, Sven Schreiber wrote:
"launch" is really intended for opening GUI applications
from within gretl.
Well I didn't necessarily want to use it as it was intended ;-)
It sounds as if MPI can do what you want here. Something like the
following:
<shell>
mpiexec -n 2 gretlmpi parallel.inp
</shell>
I was aware of MPI in principle and thought about it, but I considered
only an mpi...end mpi block. Maybe this gretlmpi thing might be an
alternative, yes, thanks for pointing it out.
cheers,
sven
10:41 a.m.
Am 07.01.2017 um 16:39 schrieb Sven Schreiber:
I was aware of MPI in principle and thought about it, but I
considered
only an mpi...end mpi block. Maybe this gretlmpi thing might be an
alternative, yes, thanks for pointing it out.
BTW, when I want to download the gretl+MPI manual I get a window with
the following error: "cURL error 60 (Peer certificate cannot be
authenticated with given CA certificates)"
I'm not behind a proxy nor do I have it set in the preferences now.
thanks,
sven
10:52 a.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
Am 07.01.2017 um 16:39 schrieb Sven Schreiber:
> I was aware of MPI in principle and thought about it, but I considered
> only an mpi...end mpi block. Maybe this gretlmpi thing might be an
> alternative, yes, thanks for pointing it out.
BTW, when I want to download the gretl+MPI manual I get a window with the
following error: "cURL error 60 (Peer certificate cannot be authenticated
with given CA certificates)"
Is that on Windows? If so, could you try replacing the bundled copy
of curl-ca-bundle.crt with
http://ricardo.ecn.wfu.edu/pub/gretl/curl-ca-bundle.crt
Maybe the certificates we're packing are out of date.
Allin
10:58 a.m.
Am 07.01.2017 um 16:52 schrieb Allin Cottrell:
On Sat, 7 Jan 2017, Sven Schreiber wrote:
>
> BTW, when I want to download the gretl+MPI manual I get a window with
> the following error: "cURL error 60 (Peer certificate cannot be
> authenticated with given CA certificates)"
Is that on Windows? If so, could you try replacing the bundled copy of
curl-ca-bundle.crt with
http://ricardo.ecn.wfu.edu/pub/gretl/curl-ca-bundle.crt
Yes, on Windows (10). But overwriting that file didn't change anything
(also not after restarting gretl).
-sven
11:16 a.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
Am 07.01.2017 um 16:52 schrieb Allin Cottrell:
> On Sat, 7 Jan 2017, Sven Schreiber wrote:
>>
>> BTW, when I want to download the gretl+MPI manual I get a window with
>> the following error: "cURL error 60 (Peer certificate cannot be
>> authenticated with given CA certificates)"
>
> Is that on Windows? If so, could you try replacing the bundled copy of
> curl-ca-bundle.crt with
>
> http://ricardo.ecn.wfu.edu/pub/gretl/curl-ca-bundle.crt
Yes, on Windows (10). But overwriting that file didn't change anything (also
not after restarting gretl).
Hmm, that's the latest version from the cURL site. Could you try one
more variant (extracted from current Firefox):
http://ricardo.ecn.wfu.edu/pub/gretl/ca-bundle.crt
(should be installed as "curl-ca-bundle.crt")
Allin
11:24 a.m.
Am 07.01.2017 um 17:16 schrieb Allin Cottrell:
Hmm, that's the latest version from the cURL site. Could you try one
more variant (extracted from current Firefox):
http://ricardo.ecn.wfu.edu/pub/gretl/ca-bundle.crt
(should be installed as "curl-ca-bundle.crt")
Just to make sure, are we talking about the top level of the gretl
install directory? (something like C:\Program Files\gretl)
Nothing changed (yesterday's snapshot). I can test a little later if
rebooting helps.
thanks,
sven
11:28 a.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
Am 07.01.2017 um 17:16 schrieb Allin Cottrell:
>
> Hmm, that's the latest version from the cURL site. Could you try one
> more variant (extracted from current Firefox):
>
> http://ricardo.ecn.wfu.edu/pub/gretl/ca-bundle.crt
>
> (should be installed as "curl-ca-bundle.crt")
Just to make sure, are we talking about the top level of the gretl install
directory? (something like C:\Program Files\gretl)
Yes, that's where it should go (there should be a file named
curl-ca-bundle.crt there already, dating from 2015).
Nothing changed (yesterday's snapshot). I can test a little later
if
rebooting helps.
Thanks. I suppose it's possible that some cache is operating such
that the updated certs are not being seen yet.
Allin
11:53 a.m.
Am 07.01.2017 um 17:28 schrieb Allin Cottrell:
On Sat, 7 Jan 2017, Sven Schreiber wrote:
> Nothing changed (yesterday's snapshot). I can test a little later if
> rebooting helps.
Thanks. I suppose it's possible that some cache is operating such that
the updated certs are not being seen yet.
Nope, no change, same error.
sorry,
sven
12:28 p.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
Am 07.01.2017 um 17:28 schrieb Allin Cottrell:
> On Sat, 7 Jan 2017, Sven Schreiber wrote:
>
>> Nothing changed (yesterday's snapshot). I can test a little later if
>> rebooting helps.
>
> Thanks. I suppose it's possible that some cache is operating such that
> the updated certs are not being seen yet.
Nope, no change, same error.
I've just put up new Windows snapshots which I think should fix this
problem (cURL error 60: Peer certificate cannot be authenticated
with known CA certificates).
Allin
11:54 a.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
Am 07.01.2017 um 16:39 schrieb Sven Schreiber:
> I was aware of MPI in principle and thought about it, but I considered
> only an mpi...end mpi block. Maybe this gretlmpi thing might be an
> alternative, yes, thanks for pointing it out.
BTW, when I want to download the gretl+MPI manual I get a window with the
following error: "cURL error 60 (Peer certificate cannot be authenticated
with given CA certificates)"
I'm not behind a proxy nor do I have it set in the preferences now.
Allin: is there a reason why the GUI client tries ricardo instead of
SourceForge?
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
12:10 p.m.
On Sat, 7 Jan 2017, Riccardo (Jack) Lucchetti wrote:
On Sat, 7 Jan 2017, Sven Schreiber wrote:
> Am 07.01.2017 um 16:39 schrieb Sven Schreiber:
>
>> I was aware of MPI in principle and thought about it, but I considered
>> only an mpi...end mpi block. Maybe this gretlmpi thing might be an
>> alternative, yes, thanks for pointing it out.
>
> BTW, when I want to download the gretl+MPI manual I get a window with the
> following error: "cURL error 60 (Peer certificate cannot be authenticated
> with given CA certificates)"
>
> I'm not behind a proxy nor do I have it set in the preferences now.
Allin: is there a reason why the GUI client tries ricardo instead of
SourceForge?
We go to sourceforge for manual files such as the one Sven was
trying to get. We go to ricardo only for function packages and gretl
databases (though it would good to shift that functionality to SF
too, if possible).
Allin
11:03 a.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
Am 07.01.2017 um 15:52 schrieb Allin Cottrell:
> It sounds as if MPI can do what you want here. Something like the
> following:
> <shell>
> mpiexec -n 2 gretlmpi parallel.inp
> </shell>
I was aware of MPI in principle and thought about it, but I considered only
an mpi...end mpi block. Maybe this gretlmpi thing might be an alternative,
yes, thanks for pointing it out.
Actually MPI may be overkill here (if you don't need any interaction
between the parallel gretl processes); you can just use the shell
for parallelism.
<hansl>
set echo off
set messages off
scalar p = scriptopt
scalar T = 100
scalar k = 3
matrix X
# conditionality here
if p == 1
X = mnormal(T, k)
elif p == 2
X = muniform(T, k)
endif
matrix XXi = inv(X'X)
string outname = sprintf("output%d.txt", p)
outfile @outname --write
print XXi
outfile --close
</hansl>
<bash>
gretlcli --scriptopt=1 -b parallel.inp &
gretlcli --scriptopt=2 -b parallel.inp &
wait
echo All done
</bash>
or
<cmd.exe>
start gretlcli --scriptopt=1 -b parallel.inp
start gretlcli --scriptopt=2 -b parallel.inp
</cmd.exe>
Allin
11:07 a.m.
Am 07.01.2017 um 17:03 schrieb Allin Cottrell:
Actually MPI may be overkill here (if you don't need any
interaction
between the parallel gretl processes); you can just use the shell for
parallelism.
<cmd.exe>
start gretlcli --scriptopt=1 -b parallel.inp
start gretlcli --scriptopt=2 -b parallel.inp
</cmd.exe>
Yeah, now you're converging exactly to what I was trying to do! Launch
several gretl processes with the help of the shell. Only that I want to
be able to do it from within gretl, hence the use of the "launch"
command in gretl/hansl.
-sven
11:30 a.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
Am 07.01.2017 um 17:03 schrieb Allin Cottrell:
> Actually MPI may be overkill here (if you don't need any interaction
> between the parallel gretl processes); you can just use the shell for
> parallelism.
> <cmd.exe>
> start gretlcli --scriptopt=1 -b parallel.inp
> start gretlcli --scriptopt=2 -b parallel.inp
> </cmd.exe>
Yeah, now you're converging exactly to what I was trying to do! Launch
several gretl processes with the help of the shell. Only that I want to be
able to do it from within gretl, hence the use of the "launch" command in
gretl/hansl.
I don't quite get why it's advantageous to do this from within
gretl, but you might try (on Windows):
<hansl>
! start gretlcli -b script.inp
</hansl>
I believe that "start" is supposed to background a task under
cmd.exe.
Allin
7:45 a.m.
Am 07.01.2017 um 17:30 schrieb Allin Cottrell:
On Sat, 7 Jan 2017, Sven Schreiber wrote:
> Yeah, now you're converging exactly to what I was trying to
do! Launch
> several gretl processes with the help of the shell. Only that I want
> to be able to do it from within gretl, hence the use of the "launch"
> command in gretl/hansl.
I don't quite get why it's advantageous to do this from within gretl,
but you might try (on Windows):
<hansl>
! start gretlcli -b script.inp
</hansl>
I believe that "start" is supposed to background a task under cmd.exe.
Yes that seems to work, including the fact that the started gretlcli has
the same CWD set, which is good.
For the same thing to work on *nixy platforms: Do I get your earlier
statement ("For "launch" ... I've now modified this to use a more
complex function that allows for such a [CWD] setting.") right in the
sense that the effect of "! start ..." on Windows can thus be obtained
on Linux with "launch ...." (in git)?
I've also experimented with MPI a little again. One problem seems to be
that it's not so easy to get shared data from before the mpi block to be
available in the mpi block scope. For example, the following gives an
error "hey undefined":
<hansl>
function void mc(scalar hey)
mpi
s = hey
end mpi
end function
mc(2)
</hansl>
Perhaps one needs a "--send-variables" option similar to --send-data for
the foreign command or to the --send-functions option from mpi itself.
Also, in the mpi documentation I came across the footnote " A possible
future extension is to support the passing of bundles between
processes." and I think that's exactly right, for my purposes just
passing scalars and matrices is not enough; especially given our adopted
style for function packages to setup a bundle initially.
Later / tomorrow I'm hopefully going to put together a toy example that
should also show why it's nice to be able to do it from within gretl.
thanks,
sven
12:37 p.m.
Wow, that is quite cool. I was looking for something like this for quite
a while but wasn't sure how to implement this. Thanks, Allin!
Artur
<bash>
gretlcli --scriptopt=1 -b parallel.inp &
gretlcli --scriptopt=2 -b parallel.inp &
wait
echo All done
</bash>
or
<cmd.exe>
start gretlcli --scriptopt=1 -b parallel.inp
start gretlcli --scriptopt=2 -b parallel.inp
</cmd.exe>
Allin
3:20 p.m.
On Sat, 7 Jan 2017, Sven Schreiber wrote:
So this is an "embarassingly trivial" type of parallelism,
where
each worker gets its own gretlcli instance/process.
As I've indicated, IMO either MPI or simple shell parallelism is the
way to go, but I've experimented with coercing "launch" into doing
what you wanted (on Linux, I'm not in a position to test in Windows
right now) and it can be done, though redirection of the child's
output doesn't seem to work so you have to use "outfile".
<parent>
print "parent started"
launch gretlcli -b taketime.inp
print "parent done"
</parent>
<child>
outfile child.out --write
print "child started"
loop 1000 -q
matrix X = mnormal(500,500)
endloop
print "child done"
outfile --close
</hansl>
If you run parent in gretl you get the "parent done" message right
away, and taketime.inp works on in the background to produce
child.out.
This forking stuff is fiendishly complicated and also differs
radically by platform, which is one reason for recommending software
that really knows what it's doing in this department, such as MPI or
the respective shells.
Allin
2800
days inactive
2802
days old
27 comments
4 participants
participants (4)
-
Allin Cottrell -
Artur Tarassow -
Riccardo (Jack) Lucchetti -
Sven Schreiber