[Gretl-devel] MPI data transfer
I've been working recently on bumping up the efficiency of MPI data
transfer.
To date transfer of matrices has been pretty fast, going directly
via memory, but bundles have been a lot slower: we took the approach
of serializing the bundle to XML at the source end, passing the XML
bytes, then deserializing at the destination end. This has been
quite reliable (modulo a few bugs that Marcin has brought to light!)
but it's inherently clunky.
Now in git I've arranged for mpibcast, mpisend and mpirecv to work
directly for bundles -- and as a by-product, enabled MPI transfer
for arrays. I've done a fair amount of testing of "direct" mpibcast
of bundles, and that's currently the default in git. The default for
mpisend/mpirecv is still the old XML method (since I just coded the
new variant today and it's barely tested) but if you're building
gretl yourself you can try out the new variants thus: look for the
definition of ALT_BUNSEND in lib/src/gretl_mpi.c and change it from
0 to 1.
(I should mention, none of the new stuff is in the Windows or Mac
snapshots just yet.)
I'll do more testing myself, but if anyone else wants to join in,
please do. My results so far are very encouraging. Here's some
comparative output from a script that transfers a bundle holding an
array of 20 big matrices:
old XML-based version:
gretlmpi 2019d-git
Current session: 2019-10-17 15:00
have 20 matrices
bcast time: 1.63798s
got 20 matrices
send/recv time: 1.59941s
diff_arrays gave 8.88178e-16
new direct version:
gretlmpi 2019d-git
Current session: 2019-10-17 15:01
have 20 matrices
bcast time: 0.0132543s
got 20 matrices
send/recv time: 0.0054283s
diff_arrays gave 0
(The non-zero "diff_arrays" value for the XML method is a trivial
result of the round-trip serialization and deserialization, but it's
nice to see it go to true zero under the direct method.)
Here's the script in question:
<hansl>
set verbose off
function scalar diff_arrays (const matrices M1, const matrices M2)
scalar maxdiff = 0
loop i=1..nelem(M1) -q
scalar d = max(abs(M1[i] - M2[i]))
if d > maxdiff
maxdiff = d
endif
endloop
return maxdiff
end function
mpi --send-functions
bundle b
if $mpirank == 0
matrices b.MM = array(20)
loop i=1..nelem(b.MM) -q
b.MM[i] = mnormal(1000,100)
endloop
printf "have %d matrices\n", nelem(b.MM)
set stopwatch
endif
mpibcast(&b)
if $mpirank == 0
printf "bcast time: %gs\n", $stopwatch
endif
set stopwatch
if $mpirank == 1
printf "got %d matrices\n", nelem(b.MM)
mpisend(b, 0)
elif $mpirank == 0
c = mpirecv(1)
printf "send/recv time: %gs\n", $stopwatch
scalar d = diff_arrays(b.MM, c.MM)
printf "diff_arrays gave %g\n", d
endif
end mpi --np=4
</hansl>
Allin