7:55 a.m.
Hi,
I face the problem with MPI in Windows: when I start processing on more
than 3 nodes I'm getting the following message (current git 64 bit
compilation + currenty MS MPI 10 library):
##################
MPI nodes: 4, OMP threads: 1
gretlmpi 2019d-git
job aborted:
[ranks] message
[0-1] terminated
[2] application aborted
aborting MPI_COMM_WORLD (comm=0x44000000), error 1, comm rank 2
[3] terminated
---- error analysis -----
[2] on KATEDRAWSB
C:\Program Files\gretl\gretlmpi aborted the job. abort code 1
---- error analysis -----
##################
The same code runs on Linux + OpenMPI even in 20 nodes (haavelmo). So
I'd like to ask how to debug MPI on Windows efficiently?
Marcin
--
Marcin Błażejowski
1:20 p.m.
On 06.10.2019 09:55, Marcin Błażejowski wrote:
Hi,
I face the problem with MPI in Windows: when I start processing on more
than 3 nodes I'm getting the following message (current git 64 bit
compilation + currenty MS MPI 10 library):
##################
MPI nodes: 4, OMP threads: 1
gretlmpi 2019d-git
job aborted:
[ranks] message
[0-1] terminated
[2] application aborted
aborting MPI_COMM_WORLD (comm=0x44000000), error 1, comm rank 2
[3] terminated
---- error analysis -----
[2] on KATEDRAWSB
C:\Program Files\gretl\gretlmpi aborted the job. abort code 1
---- error analysis -----
##################
The same code runs on Linux + OpenMPI even in 20 nodes (haavelmo). So
I'd like to ask how to debug MPI on Windows efficiently?
Marcin
The next part of the story. I've decided to test the code on Linux +
MPICH, so I compiled another gretl instance with binaries in, let's say,
'/home/marcin/gretl/2019c/bin' and:
1. Calling mpi block genereted overall problem: mpiexec was trying to
execute gretlmpi from '/usr/local/bin' instead of
'/home/marcin/gretl/2019c/bin'.
2. With symlik to '/home/marcin/gretl/2019c/bin/gretlmpi' everything
works fine.
3. It looks like with MPICH library "--quiet" flag doesn't work.
3. It seems that MPICH is faster (the same task lasts for +/- 187 sec.,
while with OpenMPI it takes +/- 230 sec.).
Marcin
--
Marcin Błażejowski
2:40 p.m.
On Sun, 6 Oct 2019, Marcin Błażejowski wrote:
On 06.10.2019 09:55, Marcin Błażejowski wrote:
> Hi,
>
> I face the problem with MPI in Windows: when I start processing on more
> than 3 nodes I'm getting the following message (current git 64 bit
> compilation + currenty MS MPI 10 library):
> ##################
> MPI nodes: 4, OMP threads: 1
> gretlmpi 2019d-git
>
> job aborted:
> [ranks] message
>
> [0-1] terminated
>
> [2] application aborted
> aborting MPI_COMM_WORLD (comm=0x44000000), error 1, comm rank 2
>
> [3] terminated
Do you have a "hosts" or "machinefile" active, perhaps?
The next part of the story. I've decided to test the code on
Linux +
MPICH, so I compiled another gretl instance with binaries in, let's say,
'/home/marcin/gretl/2019c/bin' and:
1. Calling mpi block genereted overall problem: mpiexec was trying to
execute gretlmpi from '/usr/local/bin' instead of
'/home/marcin/gretl/2019c/bin'.
That's controlled under /Tools/Preferences/General/MPI : you can
reset the path to mpiexec there.
3. It looks like with MPICH library "--quiet" flag
doesn't work.
Hmm, I'll take a look at that.
Allin
2:49 p.m.
On 06.10.2019 16:40, Allin Cottrell wrote:
On Sun, 6 Oct 2019, Marcin Błażejowski wrote:
> On 06.10.2019 09:55, Marcin Błażejowski wrote:
>> Hi,
>>
>> I face the problem with MPI in Windows: when I start processing on more
>> than 3 nodes I'm getting the following message (current git 64 bit
>> compilation + currenty MS MPI 10 library):
>> ##################
>> MPI nodes: 4, OMP threads: 1
>> gretlmpi 2019d-git
>>
>> job aborted:
>> [ranks] message
>>
>> [0-1] terminated
>>
>> [2] application aborted
>> aborting MPI_COMM_WORLD (comm=0x44000000), error 1, comm rank 2
>>
>> [3] terminated
Do you have a "hosts" or "machinefile" active, perhaps?
No,
just use 4 local phisical cores.
> The next part of the story. I've decided to test the code on Linux +
> MPICH, so I compiled another gretl instance with binaries in, let's say,
> '/home/marcin/gretl/2019c/bin' and:
>
> 1. Calling mpi block genereted overall problem: mpiexec was trying to
> execute gretlmpi from '/usr/local/bin' instead of
> '/home/marcin/gretl/2019c/bin'.
That's controlled under /Tools/Preferences/General/MPI : you can reset
the path to mpiexec there.
Allin, mpiexec (mpiexec.mpich in fact) was found end executed, but it
tried to call '/usr/local/bin/gretlmpi' (which was linked against
OpenMPI) instead of "local" '/home/marcin/gretl/2019c/bin/gretlmpi'.
Marcin
--
Marcin Błażejowski
3:38 p.m.
On Sun, 6 Oct 2019, Marcin Błażejowski wrote:
On 06.10.2019 16:40, Allin Cottrell wrote:
> On Sun, 6 Oct 2019, Marcin Błażejowski wrote:
>
>> On 06.10.2019 09:55, Marcin Błażejowski wrote:
>>> Hi,
>>>
>>> I face the problem with MPI in Windows: when I start processing on more
>>> than 3 nodes I'm getting the following message (current git 64 bit
>>> compilation + currenty MS MPI 10 library):
>>> ##################
>>> MPI nodes: 4, OMP threads: 1
>>> gretlmpi 2019d-git
>>>
>>> job aborted:
>>> [ranks] message
>>>
>>> [0-1] terminated
>>>
>>> [2] application aborted
>>> aborting MPI_COMM_WORLD (comm=0x44000000), error 1, comm rank 2
>>>
>>> [3] terminated
>
> Do you have a "hosts" or "machinefile" active, perhaps?
No, just use 4 local phisical cores.
I just tried running a trivial script with mpi block on Windows 10
with the current gretl snapshot. The machine has two physical cores,
4 threads, and I specified --np=4. The script ran without a problem,
so I don't know what's going on in your case. You might try running
your script with the --verbose flag.
>> 1. Calling mpi block genereted overall problem: mpiexec was
trying to
>> execute gretlmpi from '/usr/local/bin' instead of
>> '/home/marcin/gretl/2019c/bin'.
>
> That's controlled under /Tools/Preferences/General/MPI : you can reset
> the path to mpiexec there.
Allin, mpiexec (mpiexec.mpich in fact) was found end executed, but it
tried to call '/usr/local/bin/gretlmpi' (which was linked against
OpenMPI) instead of "local" '/home/marcin/gretl/2019c/bin/gretlmpi'.
Ah, OK. The path to the gretlmpi binary is obtained from the "Main
gretl directory" setting under /Tools/Preferences/General. If that's
/path/to/share/gretl we strip off the "share/gretl" and add
"bin/gretlmpi".
As for the --quiet flag, that's an option passed along to gretlmpi,
not aimed at mpiexec, so I don't know how there could be any
difference in result by MPI implementation.
Allin
8:01 a.m.
On 06.10.2019 17:38, Allin Cottrell wrote:
On Sun, 6 Oct 2019, Marcin Błażejowski wrote:
> On 06.10.2019 16:40, Allin Cottrell wrote:
As for the --quiet flag, that's an option passed along to gretlmpi,
not aimed at mpiexec, so I don't know how there could be any
difference in result by MPI implementation.
Ok,
I'm pretty sure that the problem is related with communication between
MPI nodes. In attachement there is an example, which:
1. On Linux with OpenMPI or MPICH works perfect.
2. On Windows 64 (I've tested 7 and 10) with current MS MPI (v10.1.1):
2.a. works perfect in case on 2 MPI nodes,
2.b. randomly works in case on 4 MPI nodes.
Switching off antivirus helped a little: processing time decreased.
Specyfying the port number (via "-p" flag) doesn't help.
Any ideas?
Marcin
--
Marcin Błażejowski
6:43 a.m.
On 08.10.2019 10:01, Marcin Błażejowski wrote:
On 06.10.2019 17:38, Allin Cottrell wrote:
> On Sun, 6 Oct 2019, Marcin Błażejowski wrote:
>
>> On 06.10.2019 16:40, Allin Cottrell wrote:
> As for the --quiet flag, that's an option passed along to gretlmpi,
> not aimed at mpiexec, so I don't know how there could be any
> difference in result by MPI implementation.
Ok,
I'm pretty sure that the problem is related with communication between
MPI nodes. In attachement there is an example, which:
I found it.
The problem is related to 'mpisend' and/or 'mpirecv' functions in case
we interchange bundles. So, in my example the following piece of code is
problematic (I commented out lines which do not matter):
<hansl>
if $mpirank != 0
Known_models_array_bundle["RES"] = MODEL.known_models_array
mpisend(Known_models_array_bundle, 0)
#delete MODEL
printf("Bundle 'Known_models_array' send to root by (%d)\n",
$mpirank)
else
#known_models_array0 = MODEL.known_models_array
loop i=1..$mpisize-1 --quiet
TMP = mpirecv(i)
known_models_array$i = TMP["RES"]
printf("Bundle 'Known_models_array' received from (%d)\n", i)
endloop
#M.StartModel_b = MODEL.StartModel_b
#M.StartModel_xnames = MODEL.StartModel_xnames
#delete MODEL
endif
</hansl>
Marcin
--
Marcin Błażejowski
5:15 p.m.
On Wed, 9 Oct 2019, Marcin Błażejowski wrote:
The problem is related to 'mpisend' and/or 'mpirecv'
functions in case
we interchange bundles. So, in my example the following piece of code is
problematic (I commented out lines which do not matter):
<hansl>
if $mpirank != 0
Known_models_array_bundle["RES"] = MODEL.known_models_array
mpisend(Known_models_array_bundle, 0)
#delete MODEL
printf("Bundle 'Known_models_array' send to root by (%d)\n",
$mpirank)
else
#known_models_array0 = MODEL.known_models_array
loop i=1..$mpisize-1 --quiet
TMP = mpirecv(i)
known_models_array$i = TMP["RES"]
printf("Bundle 'Known_models_array' received from (%d)\n", i)
endloop
#M.StartModel_b = MODEL.StartModel_b
#M.StartModel_xnames = MODEL.StartModel_xnames
#delete MODEL
endif
</hansl>
Here's a simplified variant of the above:
<hansl>
if $mpirank != 0
bundle b = defbundle("myrank", $mpirank)
mpisend(b, 0)
printf "bundle sent to root by %d\n", $mpirank
else
loop i=1..$mpisize-1 --quiet
b = mpirecv(i)
printf "bundle received from %d\n", i
endloop
endif
</hansl>
Not sure about this, but the above looks like it might be dodgy (get
out of sync somehow). You might try this variant, with a unified loop:
<hansl>
loop i=1..$mpisize-1 --quiet
if i == $mpirank
bundle b = defbundle("myrank", $mpirank)
mpisend(b, 0)
printf "bundle sent to root by %d\n", i
elif $mpirank == 0
b = mpirecv(i)
printf "bundle received from %d\n", i
endif
endloop
</hansl>
Allin
5:30 p.m.
On Wed, 9 Oct 2019, Allin Cottrell wrote:
<hansl>
if $mpirank != 0
bundle b = defbundle("myrank", $mpirank)
mpisend(b, 0)
printf "bundle sent to root by %d\n", $mpirank
else
loop i=1..$mpisize-1 --quiet
b = mpirecv(i)
printf "bundle received from %d\n", i
endloop
endif
</hansl>
Not sure about this, but the above looks like it might be dodgy (get
out of sync somehow) [...]
Hmm, actually I don't think that's issue. I'll try some
experimentation in search of a better answer.
Allin
7:17 p.m.
On Wed, 9 Oct 2019, Marcin Błażejowski wrote:
[regarding failure of gretl + MPI on MS Windows when the number of
processes exceeds some value]
The problem is related to 'mpisend' and/or 'mpirecv'
functions in case
we interchange bundles. [...]
I think this may now be fixed (in git and the current Windows
snapshots). It seems there may be a race condition in creating
temporary files via GLib on Windows that is not apparent on Linux.
Please give the current snapshot a try.
Allin
5:05 a.m.
On 09.10.2019 21:17, Allin Cottrell wrote:
On Wed, 9 Oct 2019, Marcin Błażejowski wrote:
I think this may now be fixed (in git and the current Windows
snapshots). It seems there may be a race condition in creating
temporary files via GLib on Windows that is not apparent on Linux.
Please give the current snapshot a try.
It works!!! Allin, you are the greatest!!! Thanks as usual.
Marcin
--
Marcin Błażejowski
1653
days inactive
1657
days old
10 comments
2 participants
participants (2)
-
Allin Cottrell -
Marcin Błażejowski