Hi Marcin, have you tried it? Intuitively I'd say that after the package is included the functions have been read into memory and are available. Then it should be possible to send them "anywhere" you like. But I may be missing something.

cheers, sven _______________________________________________ Gretl-devel mailing list Gretl-devel(a)lists.wfu.edu http://lists.wfu.edu/mailman/listinfo/gretl-devel

Can you tell us in what way your case differs from the above? Thanks.

foo1(

eval foo1()

end mpi --np=2

eval foo3()

Allin _______________________________________________ Gretl-devel mailing list Gretl-devel(a)lists.wfu.edu http://lists.wfu.edu/mailman/listinfo/gretl-devel

W dniu 27.06.2017 o 12:20, Marcin Błażejowski pisze: > W dniu 26.06.2017 o 02:16, Allin Cottrell pisze: >> Can you tell us in what way your case differs from the above? Thanks. > Allin, > that is not my case. I have a package send_func.gfn which has three > functions: foo1, foo2 and foo3 and the last one calls foo1 and foo2 > inside the mpi block. And it doesn't work, gratl says: Sorry for spaming, but it's quite important for me. So what do you think: will it be possible to pass private functions via mpi block?

W dniu 29.06.2017 o 14:51, Allin Cottrell pisze: > Could you send me a minimal example of what's not working, please? > Allin, just use 'send_func.gfn' package from the attachement in such a simple way: <hans> include send_func.gfn eval foo3() </hans> Let me explain what that package does. It's just the script you send three days ago, but saved in the package form organized as follows: 1. foo1() and foo(2) are private functions. 2. foo3() is a public function which calls foo1() and foo2() within the mpi block. After that gretlmpi complains on call to unknown function.

On Thu, 29 Jun 2017, Allin Cottrell wrote: > On Thu, 29 Jun 2017, Marcin Błażejowski wrote: [...] >> Let me explain what that package does. It's just the script you send >> three days ago, but saved in the package form organized as follows: >> >> 1. foo1() and foo(2) are private functions. >> 2. foo3() is a public function which calls foo1() and foo2() within the >> mpi block. After that gretlmpi complains on call to unknown function. > > Thanks, Marcin. I see that it works OK if I edit the gfn to make foo1() and > foo2() public. So I guess what's happening is just that --send-functions is > not including any private functions. I'll take a closer look... Hmm, it's not that we're excluding the private functions when we write out the functions-file to be sourced by gretlmpi. Here's my diagnosis. First, functions that are private to package X are "found" only when a public function from package X is executing. That's correct, it has to be that way. However, when you call foo1 or foo2 (private) from foo3, foo3 is executing only in the initial gretl process, it's not executing in any of the gretlmpi processes. Therefore foo1 and foo2 are "invisible" in gretlmpi. To make them visible they either need to be public, or foo3 has to be executed under mpi. In principle we could attempt a complex hack: if mpi is invoked within a packaged function F, then use some mechanism to pretend that F is executing within the subordinate gretlmpi processes. But I think this would both difficult and error prone.

W dniu 29.06.2017 o 19:28, Allin Cottrell pisze: > OK, I was too pessimistic. I think I found a nice simple way of > passing on the name of the caller function (if any) when invoking mpi. > Try current git. Sounds great!!! Thanks!!! Marcin

W dniu 30.06.2017 o 18:02, Marcin Błażejowski pisze: > > Hm, > send_function.gfn now works, but when I started testing --send-functions > in my (much bigger) BMA package I got an error: > ----- > mpiexec exited with status 256stdout: > This function needs a dataset in place > ----- I found it!!! In order to --send-functions works correctly I have to set 'Data requirement' package option to "No dataset needed". The same issue is with send_func.gfn in attachement.

W dniu 30.06.2017 o 22:54, Allin Cottrell pisze: > Hmm. So the public function in BMA requires a dataset, but the private > functions you're calling, under MPI, don't have such a requirement? Yes, that is exactly my case. > > Then I suppose it's a feasible hack to drop the dataset requirement > from the package specification, but that doesn't really seem right to > me: a package should, ideally, honestly state whether it needs a > dataset or not. Yeah, I agree at 100%. > I've just added in git, an option --send-data for the mpi-block > command, which works in the same sort of way as under "foreign" > (except that in this case we can use gretl's native gdt/gdtb formats > to send the data to gretlmpi). However, if you don't really need to > send the dataset this may just waste CPU cycles in your case. I see. > Seems to me that you're trying to use gretl's "mpi" block in a way > that was not intended. That leaves open the question: even if it was > not originally intended, should it nonetheless be supported, somehow? > Possibly, but I think it would be cleaner (if you want to support both > MPI-enabled gretl and basic gretl) if you were to branch the package > code early on, conditional on presence of MPI support: if so, do this > (go into MPI mode right away); if not, do the other. Let me explain my idea of incorporating MPI into BMA (and possibly into other packages). I start BMA in standard gretl instance and I let user to set options via GUI interface (or script it doesn't matter). Then I start MC3 which is random walk Monte Carlo, so I can't compute it in parallel (ok, there is a possibility to triger two MC3 chains in parallel but we (I mean me and Jacek) have work on this theoretically). That is why MC3 runs on just one core. But if I have to employ Gibbs sampler (inside MC3 chain) which can be computed in parallel (draws are independent) I then use MPI. After Gibbs sampler job is finished I go back to MC3 with results and I do next MC3 iteration. So public function needs dataset in place, private functions used in Gibbs sampler do not (data is mpibroadcast). Allin, maybe to avoid sending dataset over MPI world we could drop data requirement after MPI block is called I give the programmer an opportunity to decide whether send data (via --send-data flag) or not. In other words: 1. when we start package gretl respects dataset requirement, 2. when we use "--send-functions" in MPI block gretl drops this requirement and let the programmer to use (or not) "--send-data". What do you think?