On Tue, 10 Jan 2017, Sven Schreiber wrote: Hm. Even if it turns out to be an OS issue, it'd be good to delimit it properly so we know what to say to users. To be on the safe side, I've found myself relying more and more on the outfile command rather than the OS piping features. That could be due to the bread() function being invoked when the buffer is not yet committed to disk. We fixed that a few months ago in a way that I thoughtwas definitive, but it obviously isn't :) If you could provide a real test case, that would be very helpful. ------------------------------------------------------- Riccardo (Jack) Lucchetti Dipartimento di Scienze Economiche e Sociali (DiSES) Università Politecnica delle Marche (formerly known as Università di Ancona) r.lucchetti(a)univpm.it http://www2.econ.univpm.it/servizi/hpp/lucchetti -------------------------------------------------------

I like this one. IMO it's very very true. That's why I normally tend to an optimistic attitude. ------------------------------------------------------- Riccardo (Jack) Lucchetti Dipartimento di Scienze Economiche e Sociali (DiSES) Università Politecnica delle Marche (formerly known as Università di Ancona) r.lucchetti(a)univpm.it http://www2.econ.univpm.it/servizi/hpp/lucchetti -------------------------------------------------------

On Tue, 10 Jan 2017, Sven Schreiber wrote: Thanks, Sven. Are you sure you're actually getting parallel execution from your script? I'm not seeing it on Linux: adding a '&' doesn't work if the command is not executed from the actual shell. But as I mentioned you could try "launch", see http://lists.wfu.edu/pipermail/gretl-devel/2017-January/007250.html I'm attaching a variant of your script which permits comparison with MPI. Allin

Am 11.01.2017 um 07:51 schrieb Sven Schreiber: OK, attached is a version where I switched over from "! cmd c/" + output redirection to using launch with an outfile command embedded in the temp files. This is now really working in parallel, I verified with a more CPU-intensive task that actually 2 of the 4 CPU cores are then used instead of only one as before. Very interesting that the simple construction: mpi runfile = sprintf("hansltemp%d.inp", $mpirank + 1) run @runfile end mpi --np=@Nstr would work, are you sure? I would have never guessed that it's possible without mpiscatter or similar. (Apart from the fact that this code environment shows again that it's quite difficult to explain to users why they sometimes must write N and sometimes constructions like @Nstr, but that's a different topic.) thanks, sven

On Wed, 11 Jan 2017, Sven Schreiber wrote: Here's an update, now that bundles can be passed via MPI. See what you think. Reminder: this is (one version of) an apparatus which allows several gretl processes to work on a common problem but with different parameters, the parameters to be contained in bundles. <hansl name="mpibuns.inp"> # Everybody needs to know about b bundle b = null if $mpirank == 0 # Rank 0 does the parameter setup, and sends # param bundles out to the other workers scalar N = $mpisize bundles B = array(N) loop i=1..N -q b = null b.int = i b.mat = I(i+1) B[i] = b if i > 1 mpisend(b, i-1) endif endloop b = B[1] else # The other workers collect their bundles # from rank 0 b = mpirecv(0) endif # Common code block goes here: trivial in this case, # but can be as complex as you like b.mat = b.mat * b.int b.str = "done!" if $mpirank > 0 # Other workers send their results back mpisend(b, 0) else # Rank 0 collects them... B[1] = b loop i=2..N -q B[i] = mpirecv(i-1) endloop # and displays the results loop i=1..N -q printf "matrix %d\n%5g\n", i, B[i].mat printf "string %d: %s\n\n", i, B[i].str endloop endif </hansl> Execute with mpiexec -n <whatever> gretlmpi mpibuns.inp Allin

On Fri, 13 Jan 2017, Sven Schreiber wrote: Glad you like it. Why do you need them to be injected? That is, why not just do everything "inside" MPI? (You can wrap my whole script in "mpi", "end mpi" and run it in the GUI if you wish.) If, once the dust has settled, you want to inspect result-bundles in the GUI, then you could use bwrite to write them to disk (inside MPI) and bread (after MPI) to pull them back. (BTW, maybe awrite and aread for arrays would be nice to have?) A couple of quick comments: 1) You don't need the "stillworking" apparatus with MPI. All the relevant data-transfer functions are "blocking". This means that while you might have got stillworking == 1 in the original version of your script (one or more processes not finished yet), it can't happen under MPI: either (a) you have all the results or (b) one or more of the processes has failed in some way and therefore the whole MPI section has automatically errored out or (c) the MPI section is still running so the "stillworking" check is not reached yet. 2) I think you may find it's much nicer if you can put the common working code block into the MPI section. That way you just have to write hansl, not squirt it though printf as in printf "printf \"Where TF am I?\\n\"\n" There's some of that sort of thing in gretl_foreign.c (generating startup code for Octave, R, etc.), and it was not much fun to write! Allin

On Fri, 13 Jan 2017, Sven Schreiber wrote: Thanks, I now have a clearer picture of what you're aiming to do, and it's quite interesting. I'm attaching a new variant of my own. I've cut back on printing, error checking and options to try to expose the overall structure as clearly as possible. Besides that there are two main points that you might like to consider. 1) MPI stuff: you don't need to write out N temporary script files, one will do nicely. The only rank-specific point in the script is the name of the bundle to read/write and that can be constructed conditional on $mpirank. Also, echo and messages are off by default in an MPI block; there's no need to turn them off explicitly. 2) Per-process output files: I think that for most users most of the time it would be more user-friendly to gather the content of these files and consolidate the output, so that's what I've done. I guess one could have an option switch in this regard. Should do, yes. Point taken, in regard to the "boilerplate" code inside your function. But what about the user code? That has to be supplied as a string, so the user must write hansl-generating hansl. Easy enough in a trivial case but gets nasty if you're using printf or sprintf. I haven't done anything about this in my current variant, but if it were my package I'd probably want to give the option of supplying the user-code via the name of a script file rather than a hansl string. Last small point: my variant cleans up its temporary files, if we don't want to preserve tham for debugging purposes. Allin

On Sun, 15 Jan 2017, Sven Schreiber wrote: Perhaps, but I could see a case for printing stuff, depending on the character of the user's code. Yes, that would work. Not directly, but it can be done: <hansl> function scalar shell_ok (void) outfile tmpbuf --buffer set # read the "set" variables outfile --close string s = strstr(tmpbuf, "shell_ok") + 10 return atof(s) end function </hansl> Allin