11:22 a.m.
Hi, here's a question about what is really affected by setting the
number of threads used by OpenMP via an environment variable and/or via
the option to an MPI block (as explained in the gretl + MPI documentation).
My test script has an MPI block as its main part which calls a certain
worker function. Basically it's a VAR bootstrap, so making heavy use of
mnormal(), varsimul() and mols().
The MPI block is specified with --omp-threads=1. I thought that means
that no multithreading is done at the OpenMP level.
My blas is openblas and shows up with the option blas_parallel = "OpenMP".
However: When I run this script (on a Jan-11 Windows snapshot) with the
environment set to OMP_NUM_THREADS=1 it takes about 1/3 _less_ time than
with OMP_NUM_THREADS=4, on a 4-physical-cores CPU. This happens for any
variation of the numbers of parallel MPI processes.
OK, so given our recent discussion about the problems of openblas
multithreading, the ranking of the timings is maybe not so surprising
anymore. But: why does OMP_NUM_THREADS=1 have an additional effect over
--omp-threads=1 when everything happens inside the MPI block ?
I hope the problem was clearly explained.
thanks
sven
11:40 a.m.
On 16.01.2020 12:22, Sven Schreiber wrote:
Hi, here's a question about what is really affected by setting
the
number of threads used by OpenMP via an environment variable and/or via
the option to an MPI block (as explained in the gretl + MPI
documentation).
By setting "--omp-threads=1" in MPI block you implicit define "set
omp_num_threads 1" in 'gretltmp-mpi.inp' script which is ran by all
gretlmpi executables.
But the fact is that in many cases some routines runs in parallel via
OpenMP no matter what is the value of "omp_num_threads". I remember that
2-3 years ago Jack explained me that some algebra stuff lauch OpenMP
automagically.
Regards,
Marcin
--
Marcin Błażejowski
11:46 a.m.
Am 16.01.2020 um 12:40 schrieb Marcin Błażejowski:
On 16.01.2020 12:22, Sven Schreiber wrote:
> Hi, here's a question about what is really affected by setting the
> number of threads used by OpenMP via an environment variable and/or via
> the option to an MPI block (as explained in the gretl + MPI
> documentation).
By setting "--omp-threads=1" in MPI block you implicit define "set
omp_num_threads 1" in 'gretltmp-mpi.inp' script which is ran by all
gretlmpi executables.
Yes, that's what I thought.
But the fact is that in many cases some routines runs in parallel
via
OpenMP no matter what is the value of "omp_num_threads". I remember that
2-3 years ago Jack explained me that some algebra stuff lauch OpenMP
automagically.
Aha, that would explain it. But the question is, is it a bug? I mean if
we want to suppress multithreading, we want to do that also in the
openblas context, no?
thanks Marcin,
sven
3:13 p.m.
On 16.01.2020 12:46, Sven Schreiber wrote:
> But the fact is that in many cases some routines runs in parallel
via
> OpenMP no matter what is the value of "omp_num_threads". I remember that
> 2-3 years ago Jack explained me that some algebra stuff lauch OpenMP
> automagically.
Aha, that would explain it. But the question is, is it a bug? I mean if
we want to suppress multithreading, we want to do that also in the
openblas context, no?
I believe, yes.
Cheers,
Marcin
--
Marcin Błażejowski
3:19 p.m.
Am 16.01.2020 um 16:13 schrieb Marcin Błażejowski:
On 16.01.2020 12:46, Sven Schreiber wrote:
>> But the fact is that in many cases some routines runs in parallel via
>> OpenMP no matter what is the value of "omp_num_threads". I remember
that
>> 2-3 years ago Jack explained me that some algebra stuff lauch OpenMP
>> automagically.
> Aha, that would explain it. But the question is, is it a bug? I mean if
> we want to suppress multithreading, we want to do that also in the
> openblas context, no?
I believe, yes.
BTW, this here might also be relevant:
https://github.com/xianyi/OpenBLAS/issues/1653
Using the settings:
OMP_NUM_THREADS 4
OMP_PLACES "{0,1,2,3}"
OMP_PROC_BIND spread
If I understand correctly, just limiting OMP_NUM_THREADS alone is in
general no guarantee that hyperthreading is prevented.
Whether that makes a practical difference, I don't know (yet).
cheers
sven
4:42 p.m.
On Thu, 16 Jan 2020, Sven Schreiber wrote:
Am 16.01.2020 um 16:13 schrieb Marcin Błażejowski:
> On 16.01.2020 12:46, Sven Schreiber wrote:
> > > But the fact is that in many cases some routines runs in parallel via
> > > OpenMP no matter what is the value of "omp_num_threads". I
remember that
> > > 2-3 years ago Jack explained me that some algebra stuff lauch OpenMP
> > > automagically.
> > Aha, that would explain it. But the question is, is it a bug? I mean if
> > we want to suppress multithreading, we want to do that also in the
> > openblas context, no?
> I believe, yes.
I can't replicate the problem Sven describes, either on Linux or on
Windows 10. In each case it seems that the gretl mpi-block option
--omp-threads successfully controls the number of OMP threads, taking
precedence over a setting of OMP_NUM_THREADS in the environment.
When the user specifies omp-threads, libgretl calls the libgomp
function omp_set_num_threads(), and as I understand it that should
limit the number of threads used in calls to OpenMP from libgretl.
In light of Sven's account I wondered whether this libgomp-level
setting was not "getting through" to the openblas library, but in my
testing it appears to be working fine. Nonetheless, I've added a call
to openblas_set_num_threads() just in case that makes a difference on
some systems. (Well, it presumably _will_ make a difference on systems
where openblas uses raw pthreads rather than OMP, but that's not the
case in our packages for Windows.)
Here's the test I was running. It relies on the fact that our old svd
code ran significantly faster without multi-threading, so
GRETL_OLD_SVD=1 should be in the environment. I collected timings for
various values of OMP_NUM_THREADS and --omp-threads and found that
--omp-threads was the controlling factor.
<hansl>
set verbose off
mpi
set seed 987654
matrix U
matrix V
k = 40
N = floor(5000000/k^2)
matrix X = mnormal(110,k)
set stopwatch
loop N -q
matrix s = svd(X, &U, &V)
endloop
printf "k = %d, elapsed: %#gs\n", k, $stopwatch
end mpi --np=2 --omp-threads=1 # or increase
</hansl>
Allin
6:50 p.m.
Am 16.01.2020 um 17:42 schrieb Allin Cottrell:
I can't replicate the problem Sven describes, either on Linux or
on
Windows 10.
...
GRETL_OLD_SVD=1 should be in the environment. I collected timings
for
various values of OMP_NUM_THREADS and --omp-threads and found that
--omp-threads was the controlling factor.
Here's what I'm getting with the
latest snapshot (slightly edited script
attached, remember number of MPI processes is fixed at 2 there), and I'm
running this with gretlcli.exe. This is Windows 10 on an i7-8550U (4
cores plus HT; not the same machine when I started this thread - nomen
est omen!).
* Indeed when the --omp-threads=1 option is used, the setting of
OMP_NUM_THREADS makes no difference. This is different to what I had
before, but the machine and the snapshot have changed. (The different
application doesn't matter.) I will re-check on the other machine later.
* But in general the interaction nevertheless seems more complicated,
not just a matter of what takes precedence. I'm pasting the results below.
thanks
sven
--------------------
- local setting 2, environment setting going from 1 to 2 dramatically
increases the time:
GRETL_OLD_SVD is 1
OMP_NUM_THREADS is 1
--omp-threads option is 2
Entering MPI real-time read loop
gretlmpi 2020a-git
Aktuelle Sitzung: 2020-01-16 19:26
k = 40, elapsed: 1.83464s
k = 40, elapsed: 1.94281s
GRETL_OLD_SVD is 1
OMP_NUM_THREADS is 2
--omp-threads option is 2
Entering MPI real-time read loop
gretlmpi 2020a-git
Aktuelle Sitzung: 2020-01-16 19:29
k = 40, elapsed: 10.7115s
k = 40, elapsed: 10.8237s
- local setting 4, environment going from 1 to 2 even more so:
GRETL_OLD_SVD is 1
OMP_NUM_THREADS is 1
--omp-threads option is 4
Entering MPI real-time read loop
gretlmpi 2020a-git
Aktuelle Sitzung: 2020-01-16 19:27
k = 40, elapsed: 1.82055s
k = 40, elapsed: 1.93062s
GRETL_OLD_SVD is 1
OMP_NUM_THREADS is 2
--omp-threads option is 4
Entering MPI real-time read loop
gretlmpi 2020a-git
Aktuelle Sitzung: 2020-01-16 19:28
k = 40, elapsed: 14.9104s
k = 40, elapsed: 15.0594s
8:34 p.m.
Am 16.01.2020 um 19:50 schrieb Sven Schreiber:
Am 16.01.2020 um 17:42 schrieb Allin Cottrell:
> I can't replicate the problem Sven describes, either on Linux or on
> Windows 10.
* Indeed when the --omp-threads=1 option is used, the setting of
OMP_NUM_THREADS makes no difference. This is different to what I had
before, but the machine and the snapshot have changed. (The different
application doesn't matter.) I will re-check on the other machine later.
I now think the differences I saw were spurious, perhaps due to some
additionally running background process or whatever. Some such
fluctuations could be observed, but the "best" runtimes in that
application do indeed not seem to be affected by OMP_NUM_THREADS.
Sorry for the noise there.
The slowdowns of the previous message with Allin's script (e.g.
--omp-threads=2 and OMP_NUM_THREADS=2) were much more drastic, however.
thanks
9:18 p.m.
On Thu, 16 Jan 2020, Sven Schreiber wrote:
Am 16.01.2020 um 19:50 schrieb Sven Schreiber:
> Am 16.01.2020 um 17:42 schrieb Allin Cottrell:
> > I can't replicate the problem Sven describes, either on Linux or on
> > Windows 10.
> * Indeed when the --omp-threads=1 option is used, the setting of
> OMP_NUM_THREADS makes no difference. This is different to what I had
> before, but the machine and the snapshot have changed. (The different
> application doesn't matter.) I will re-check on the other machine later.
I now think the differences I saw were spurious, perhaps due to some
additionally running background process or whatever. Some such
fluctuations could be observed, but the "best" runtimes in that
application do indeed not seem to be affected by OMP_NUM_THREADS.
Sorry for the noise there.
The slowdowns of the previous message with Allin's script (e.g.
--omp-threads=2 and OMP_NUM_THREADS=2) were much more drastic, however.
I can't replicate those slowdowns on Linux (for each --omp-threads
case, times are invariant with respect to the OMP_NUM_THREADS setting)
but I'll try on Windows tomorrow.
Allin
10:09 p.m.
Am 16.01.2020 um 22:18 schrieb Allin Cottrell:
On Thu, 16 Jan 2020, Sven Schreiber wrote:
> Am 16.01.2020 um 19:50 schrieb Sven Schreiber:
> The slowdowns of the previous message with Allin's script
(e.g.
> --omp-threads=2 and OMP_NUM_THREADS=2) were much more drastic, however.
I can't replicate those slowdowns on Linux (for each --omp-threads
case, times are invariant with respect to the OMP_NUM_THREADS setting)
but I'll try on Windows tomorrow.
I'm seeing this on another machine as well, timings going from roughly
2.5s to 8.5s when OMP_NUM_THREADS=1 is changed to OMP_NUM_THREADS=2 (and
--omp-threads=2). This was still the Jan-11th snapshot.
Then I have upgraded to the latest snapshot of "just now". There I don't
see a noticeable difference anymore - but that's because both variants
are now slow!
Like this:
GRETL_OLD_SVD is 1
OMP_NUM_THREADS is 1
--omp-threads option is 2
Entering MPI real-time read loop
gretlmpi 2020a-git
Aktuelle Sitzung: 2020-01-16 23:08
k = 40, elapsed: 8.36354s
k = 40, elapsed: 8.50609s
Hm..
sven
12:13 a.m.
On Thu, 16 Jan 2020, Sven Schreiber wrote:
Am 16.01.2020 um 22:18 schrieb Allin Cottrell:
> On Thu, 16 Jan 2020, Sven Schreiber wrote:
>
>> Am 16.01.2020 um 19:50 schrieb Sven Schreiber:
>> The slowdowns of the previous message with Allin's script (e.g.
>> --omp-threads=2 and OMP_NUM_THREADS=2) were much more drastic, however.
>
> I can't replicate those slowdowns on Linux (for each --omp-threads
> case, times are invariant with respect to the OMP_NUM_THREADS setting)
> but I'll try on Windows tomorrow.
I'm seeing this on another machine as well, timings going from roughly
2.5s to 8.5s when OMP_NUM_THREADS=1 is changed to OMP_NUM_THREADS=2 (and
--omp-threads=2). This was still the Jan-11th snapshot.
Then I have upgraded to the latest snapshot of "just now". There I don't
see a noticeable difference anymore - but that's because both variants
are now slow!
Like this:
GRETL_OLD_SVD is 1
OMP_NUM_THREADS is 1
--omp-threads option is 2
Entering MPI real-time read loop
gretlmpi 2020a-git
Aktuelle Sitzung: 2020-01-16 23:08
k = 40, elapsed: 8.36354s
k = 40, elapsed: 8.50609s
On the maintained hypothesis that single-threaded is fastest for
DGESVD, I think the interpretation has to be the following (though I
can't really say I understand it yet):
* Under the old scheme, OMP_NUM_THREADS=1 was overriding the
omp-threads setting (because OMP_NUM_THREADS got higher priority in
general, or because the minimum value ruled, or because omp-threads
was somehow just ignored at openblas level).
* Under the new scheme omp-threads rules, so you'll get slow results
with that set to 2 regardless of the OMP_NUM_THREADS setting --
which is, I think, as it should be.
If that's right, the only real cause for concern would be if you
now (but how?!) get slow performance with --omp-threads=1.
Allin
8:50 a.m.
Am 17.01.2020 um 01:13 schrieb Allin Cottrell:
On Thu, 16 Jan 2020, Sven Schreiber wrote:
> Then I have upgraded to the latest snapshot of "just
now". There I don't
> see a noticeable difference anymore - but that's because both variants
> are now slow!
> Like this:
> GRETL_OLD_SVD is 1
> OMP_NUM_THREADS is 1
> --omp-threads option is 2
On the maintained hypothesis that single-threaded is fastest for
DGESVD,
... for the given dimension 110 x 40 ...
* Under the new scheme omp-threads rules, so you'll get slow
results
with that set to 2 regardless of the OMP_NUM_THREADS setting -- which
is, I think, as it should be.
Does the "new" scheme cover the snapshot as of 14 hours ago? Because
there I had posted the case with "--omp-threads option is 2" where
OMP_NUM_THREADS did make a difference.
I guess I'm being confusing because I've also switched back and forth
between two machines. One is Ivybridge (pre-Haswell) and has no HT, the
other is Kabylake (post-Haswell) or so with HT. The thing from 14 hours
ago was with the latter; maybe HT has an influence there.
(I could try the additional environment settings I posted some messages
back to suppress HT completely, but haven't done that yet.)
If that's right, the only real cause for concern would be if you
now
(but how?!) get slow performance with --omp-threads=1.
No, I don't, everything's fine with --omp-threads=1.
thanks
sven
9:11 a.m.
Am 16.01.2020 um 17:42 schrieb Allin Cottrell:
In light of Sven's account I wondered whether this libgomp-level
setting was not "getting through" to the openblas library, but in my
testing it appears to be working fine. Nonetheless, I've added a call
to openblas_set_num_threads() just in case that makes a difference on
some systems.
openblas_set_num_threads(1) is indeed one of the recommended ways
(https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded)
BTW, slightly OT, but since you upgraded OpenBLAS to 0.3.7, the page
above also says:
"when OpenBLAS was built single-threaded by USE_THREAD=0 ... it is vital
to also specify USE_LOCKING=1 (introduced with OpenBLAS 0.3.7)"
I think normally gretl doesn't use single-threaded openblas, but it
sounds relevant for the exceptional cases where it wants to do that.
cheers
sven
9:22 a.m.
Am 17.01.2020 um 10:11 schrieb Sven Schreiber:
Am 16.01.2020 um 17:42 schrieb Allin Cottrell:
> In light of Sven's account I wondered whether this libgomp-level
> setting was not "getting through" to the openblas library, but in my
> testing it appears to be working fine. Nonetheless, I've added a call
> to openblas_set_num_threads() just in case that makes a difference on
> some systems.
openblas_set_num_threads(1) is indeed one of the recommended ways
(https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded)
P.S. On the main page https://github.com/xianyi/OpenBLAS it says about
environment settings:
The priorities are OPENBLAS_NUM_THREADS > GOTO_NUM_THREADS >
OMP_NUM_THREADS.
-s
5:53 p.m.
On Fri, 17 Jan 2020, Sven Schreiber wrote:
Am 17.01.2020 um 10:11 schrieb Sven Schreiber:
> Am 16.01.2020 um 17:42 schrieb Allin Cottrell:
>
> > In light of Sven's account I wondered whether this libgomp-level
> > setting was not "getting through" to the openblas library, but in my
> > testing it appears to be working fine. Nonetheless, I've added a call
> > to openblas_set_num_threads() just in case that makes a difference on
> > some systems.
>
> openblas_set_num_threads(1) is indeed one of the recommended ways
> (https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded)
P.S. On the main page https://github.com/xianyi/OpenBLAS it says about
environment settings:
The priorities are OPENBLAS_NUM_THREADS > GOTO_NUM_THREADS >
OMP_NUM_THREADS.
If openblas is built with OpenMP support, only the last of these
environment variables has any effect.
Allin
1:30 p.m.
On Fri, 17 Jan 2020, Sven Schreiber wrote:
Am 16.01.2020 um 17:42 schrieb Allin Cottrell:
> In light of Sven's account I wondered whether this libgomp-level
> setting was not "getting through" to the openblas library, but in my
> testing it appears to be working fine. Nonetheless, I've added a call
> to openblas_set_num_threads() just in case that makes a difference on
> some systems.
openblas_set_num_threads(1) is indeed one of the recommended ways
(https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded)
BTW, slightly OT, but since you upgraded OpenBLAS to 0.3.7, the page
above also says:
"when OpenBLAS was built single-threaded by USE_THREAD=0 ... it is vital
to also specify USE_LOCKING=1 (introduced with OpenBLAS 0.3.7)"
I think normally gretl doesn't use single-threaded openblas, but it
sounds relevant for the exceptional cases where it wants to do that.
That's not an issue for us: "built single-threaded" is a compile-time
thing. We (I) compile openblas only for our Windows packages, and in
that context we're never going to supply a single-threaded build --
although the library will _run_ single-threaded is there's only one
processor on the user's system.
On Linux, it's up to the distro to ensure that a single-threaded
version of openblas (if relevant) is done right. And on Mac we use the
built-in Accelerate Framework instead.
Allin
3:20 p.m.
On Thu, 16 Jan 2020, Sven Schreiber wrote:
Hi, here's a question about what is really affected by setting
the
number of threads used by OpenMP via an environment variable and/or via
the option to an MPI block (as explained in the gretl + MPI documentation).
My test script has an MPI block as its main part which calls a certain
worker function. Basically it's a VAR bootstrap, so making heavy use of
mnormal(), varsimul() and mols().
The MPI block is specified with --omp-threads=1. I thought that means
that no multithreading is done at the OpenMP level.
My blas is openblas and shows up with the option blas_parallel = "OpenMP".
However: When I run this script (on a Jan-11 Windows snapshot) with the
environment set to OMP_NUM_THREADS=1 it takes about 1/3 _less_ time than
with OMP_NUM_THREADS=4, on a 4-physical-cores CPU. This happens for any
variation of the numbers of parallel MPI processes.
OK, so given our recent discussion about the problems of openblas
multithreading, the ranking of the timings is maybe not so surprising
anymore. But: why does OMP_NUM_THREADS=1 have an additional effect over
--omp-threads=1 when everything happens inside the MPI block ?
I hope the problem was clearly explained.
Just about -- but there's one thing I need to know: when the script is
taking a relatively long time, do you or don't you have
OMP_NUM_THREADS=4 set in the environment?
Allin
3:24 p.m.
Am 16.01.2020 um 16:20 schrieb Allin Cottrell:
On Thu, 16 Jan 2020, Sven Schreiber wrote:
> OK, so given our recent discussion about the problems of openblas
> multithreading, the ranking of the timings is maybe not so surprising
> anymore. But: why does OMP_NUM_THREADS=1 have an additional effect over
> --omp-threads=1 when everything happens inside the MPI block ?
>
> I hope the problem was clearly explained.
Just about -- but there's one thing I need to know: when the script is
taking a relatively long time, do you or don't you have
OMP_NUM_THREADS=4 set in the environment?
Yes I do. Do you mean it takes precedence over the "inner" --omp-threads
setting?
thanks
sven
3:28 p.m.
On Thu, 16 Jan 2020, Sven Schreiber wrote:
Am 16.01.2020 um 16:20 schrieb Allin Cottrell:
> On Thu, 16 Jan 2020, Sven Schreiber wrote:
>
> > OK, so given our recent discussion about the problems of openblas
> > multithreading, the ranking of the timings is maybe not so surprising
> > anymore. But: why does OMP_NUM_THREADS=1 have an additional effect over
> > --omp-threads=1 when everything happens inside the MPI block ?
> >
> > I hope the problem was clearly explained.
> Just about -- but there's one thing I need to know: when the script is
> taking a relatively long time, do you or don't you have
> OMP_NUM_THREADS=4 set in the environment?
>
Yes I do. Do you mean it takes precedence over the "inner" --omp-threads
setting?
I don't think so but I'm not sure. Just checked and on Linux, at any
rate, the --omp-threads setting wins.
Allin
1801
days inactive
1802
days old
18 comments
3 participants
participants (3)
-
Allin Cottrell -
Marcin Błażejowski -
Sven Schreiber