Hi, what such an error means (haavelmo machine, MPI in 20 threads):

On 13.08.2019 12:09, Riccardo (Jack) Lucchetti wrote: > On Tue, 13 Aug 2019, Marcin Błażejowski wrote: > >> Hi, >> >> what such an error means (haavelmo machine, MPI in 20 threads): > > Just for the record: "haavelmo" is a server at UnivPM I gave Marcin > access to for the purpose of testing MPI-related stuff (In case you > people were wondering!). It looks like the problem with concurent access to the file. I solved the issue, but the solution is quite numb: ########################## loop i=0..$mpisize-1 --quiet     if ($mpirank == i)         MODEL = bread("MODEL")     else         sleep(1)     endif endloop ##########################

On 13.08.2019 19:22, Allin Cottrell wrote: > On Tue, 13 Aug 2019, Marcin Błażejowski wrote: > >> It looks like the problem with concurent access to the file. I solved >> the issue, but the solution is quite numb: >> ########################## >> loop i=0..$mpisize-1 --quiet >>     if ($mpirank == i) >>         MODEL = bread("MODEL") >>     else >>         sleep(1) >>     endif >> endloop >> ########################## > > Possibly relevant questions: Is the bundle file gzipped or just plain > XML? How/when is the bundle file created? Allin, this is a plain XML created by bwrite() few lines before bread() is used:

On 13.08.2019 21:28, Allin Cottrell wrote: > On Tue, 13 Aug 2019, Marcin Błażejowski wrote: > >> On 13.08.2019 19:22, Allin Cottrell wrote: >>> On Tue, 13 Aug 2019, Marcin Błażejowski wrote: >>> >>>> It looks like the problem with concurent access to the file. I solved >>>> the issue, but the solution is quite numb: >>>> ########################## >>>> loop i=0..$mpisize-1 --quiet >>>>     if ($mpirank == i) >>>>         MODEL = bread("MODEL") >>>>     else >>>>         sleep(1) >>>>     endif >>>> endloop >>>> ########################## >>> Possibly relevant questions: Is the bundle file gzipped or just plain >>> XML? How/when is the bundle file created? >> Allin, >> >> this is a plain XML created by bwrite() few lines before bread() is >> used: > OK. I assume the bundle is being written by one process, to be read by > all, or all the others? > > If so, you probably need to add an MPI barrier, otherwise some > processes may try to start reading the file before the writer has > finished writing it. > > We don't have a simple MPI barrier as such in our hansl implementation > of MPI (maybe we should have such), but you could > just broadcast a scalar from the process that's writing the file. > Something like: > > <hansl> > # write bundle HERE > scalar bundle_ready > if $mpirank == 0 # or whoever is writing >     bundle_ready = 1 > endif > mpibcast(&bundle_ready) > if bundle_ready >     # read bundle HERE > endif > <hansl> Ok, thanks Allin. In my case "# write bundle HERE" is done before MPI block, but I understand it doesn't matter?

On Wed, 14 Aug 2019, Marcin Błażejowski wrote: > Hmm, the workaround doesn't work. Ah, I think see what the real problem is (if I'm right, it's specific to bundles containing big matrices), and I'm working on a fix.

> BTW: can I suppress messages from MPI nodes about loading dataset when > MPI '--send-data' flag is used? I don't think so, at present, but it ought to be possible.

On Wed, 14 Aug 2019, Marcin Błażejowski wrote: > On 14.08.2019 16:05, Allin Cottrell wrote: >> On Wed, 14 Aug 2019, Marcin Błażejowski wrote: >> >>> Hmm, the workaround doesn't work. >> Ah, I think see what the real problem is (if I'm right, it's specific >> to bundles containing big matrices), and I'm working on a fix. > I see. I'm sending the bundle I use on my 4-cores machine, maybe it'll > be useful. >>> BTW: can I suppress messages from MPI nodes about loading dataset when >>> MPI '--send-data' flag is used? >> I don't think so, at present, but it ought to be possible. > That is great!!! I'm glad someone is exercising our MPI code!

Anyway, those two points should now be fixed in git.

On 14.08.2019 16:49, Marcin Błażejowski wrote: > On 14.08.2019 16:43, Allin Cottrell wrote: > >> Anyway, those two points >> should now be fixed in git. > Thanks again!!! I will try in tommorow in the morning!!! And it works perfect!!!