7:12 a.m.
Hi Allin and Jack,
I was trying to run a fixed effect logit model on the wagepan dataset using Jack’s felogit
package (version 1.8), which I have installed as a menu item to use in the GUI.
After I provided the binary variable and a list of regressors in the dialog box, if I also
selected the time dummies, gretl crashed once I hit ok. Without the time dummies, it
worked fine.
I then run the same felogit package in a script, it worked fine without the time dummies,
and crashed with time dummies are included.
Best,
Fred
Attached is the crash report using GUI.
9:07 a.m.
Am 25.10.2020 um 08:12 schrieb Fred Engst:
Hi Allin and Jack,
I was trying to run a fixed effect logit model on the wagepan dataset using Jack’s
felogit package (version 1.8), which I have installed as a menu item to use in the GUI.
After I provided the binary variable and a list of regressors in the dialog box, if I
also selected the time dummies, gretl crashed once I hit ok. Without the time dummies, it
worked fine.
I then run the same felogit package in a script, it worked fine without the time dummies,
and crashed with time dummies are included.
Best,
Fred
Attached is the crash report using GUI.
THat's a very thorough report, thanks!
I cannot confirm this on Windows, which doesn't mean it's not real on
Mac, though. Maybe you could be more specific which data you used.
This is just speculation, but I find the portions with
libsystem_pthread.dylib in the different threads suspicious. Could there
be some confusion between openmp and pthreads going on? But perhaps the
other threads in this report do not play a role at all, I don't know
enough about this kind of output on Mac, sorry.
cheers
sven
10:34 a.m.
On Sun, 25 Oct 2020, Fred Engst wrote:
Hi Allin and Jack,
I was trying to run a fixed effect logit model on the wagepan dataset using Jack’s
felogit package (version
1.8), which I have installed as a menu item to use in the GUI.
After I provided the binary variable and a list of regressors in the dialog box, if I
also selected the time
dummies, gretl crashed once I hit ok. Without the time dummies, it worked fine.
I then run the same felogit package in a script, it worked fine without the time dummies,
and crashed with time
dummies are included.
My guess is that there is some problem in the OSX BLAS library. The crash
seems to happen in the context of the check we run, prior to estimation,
on the information matrix to prune out collinear regressors.
I used Wooldridge's wagepan dataset (I believe this is what you are
referring to) and ran something like
<hansl>
include felogit.gfn
open wagepan.dta --quiet
setobs nr year --panel-vars
list X = black exper hisp hours married
bundle b = felogit(union, X, 2, 1)
</hansl>
The specification above is designed to trigger the exclusion of some
explanatory variables. What I'm getting is
<output>
--- Iteration
1: Criterion -730.394459351 (step 1, norm 1.61e+00)
2: Criterion -730.388318181 (step 1, norm 2.67e-01)
3: Criterion -730.388318167 (step 1, norm 9.46e-03)
--- FINAL VALUES:
Criterion -730.388318167 (step 1, norm 1.57e-05)
Fixed effects logit: 246 units (out of 545)
1968 observations (out of 4360)
Tmin = 8, Tmax = 8
Dependent variable = union
coefficient std. error z p-value
----------------------------------------------------------
exper 0.0131227 0.0320862 0.4090 0.6826
hours -0.000252063 0.000124456 -2.025 0.0428 **
married 0.302405 0.170595 1.773 0.0763 *
Dtime_2 -0.0438914 0.192428 -0.2281 0.8196
Dtime_3 0.0170427 0.182519 0.09337 0.9256
Dtime_4 -0.131438 0.179846 -0.7308 0.4649
Dtime_5 -0.0854543 0.178726 -0.4781 0.6326
Dtime_6 -0.424490 0.186375 -2.278 0.0227 **
Dtime_7 -0.597989 0.196524 -3.043 0.0023 ***
Log-likelihood -730.388 AIC 1478.777
BIC 1529.040 HQC 1497.247
Wald test = 20.3521 (9 df); p-value = 0.0158594
</output>
Do you get the same output on your PC? Thanks.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
1:51 p.m.
On Sun, 25 Oct 2020, Riccardo (Jack) Lucchetti wrote:
On Sun, 25 Oct 2020, Fred Engst wrote:
> Hi Allin and Jack,
> I was trying to run a fixed effect logit model on the wagepan dataset using
> Jack’s felogit package (version
> 1.8), which I have installed as a menu item to use in the GUI.
> After I provided the binary variable and a list of regressors in the dialog
> box, if I also selected the time
> dummies, gretl crashed once I hit ok. Without the time dummies, it worked
> fine.
> I then run the same felogit package in a script, it worked fine without the
> time dummies, and crashed with time
> dummies are included.
My guess is that there is some problem in the OSX BLAS library. The crash
seems to happen in the context of the check we run, prior to estimation, on
the information matrix to prune out collinear regressors.
Yes. The crash occurs in the LAPACK function DSYEVR, called by
gretl_symmetric_matrix_eigenvals(). It seems that either we're
passing bad data to DSYEVR or there's a bug in the implementation of
that function in Apple's Accelerate Framework.
I don't think gretl is passing bad data to DSYEVR -- at least
valgrind doesn't detect any problem.
There's a slower but perhaps more robust LAPACK function, DSYEV,
that can be called for this purpose -- we use it if the matrix in
question is small or if GRETL_OLD_EV is set in the environment. So
an interesting test on macOS would be to set GRETL_OLD_EV=1 and see
if the problem persists.
Allin
3:04 p.m.
On Sun, 25 Oct 2020, Allin Cottrell wrote:
So an interesting test on macOS would be to set GRETL_OLD_EV=1 and
see if the problem persists.
I just tried on macOS 10.15.6: I got the crash that Fred reported,
but when I set GRETL_OLD_EV=1 there was no crash.
It turns out this is an Apple bug: there are two arguments to DSYEVR
that are not supposed to be referenced when the function is called
in the mode gretl uses (according to the netlib LAPACK docs). So we
were just passing NULL for those arguments. But apparently Apple's
DSYEVR does reference them, hence a crash on dereferencing a null
pointer. In git (snapshots to follow) we now pass pointers to dummy
arguments in those slots as a workaround.
Allin
4:04 p.m.
Am 25.10.2020 um 16:04 schrieb Allin Cottrell:
On Sun, 25 Oct 2020, Allin Cottrell wrote:
> So an interesting test on macOS would be to set GRETL_OLD_EV=1 and see
> if the problem persists.
I just tried on macOS 10.15.6: I got the crash that Fred reported, but
when I set GRETL_OLD_EV=1 there was no crash.
It turns out this is an Apple bug: there are two arguments to DSYEVR
that are not supposed to be referenced when the function is called in
the mode gretl uses (according to the netlib LAPACK docs). So we were
just passing NULL for those arguments. But apparently Apple's DSYEVR
does reference them, hence a crash on dereferencing a null pointer. In
git (snapshots to follow) we now pass pointers to dummy arguments in
those slots as a workaround.
Do you have any link to the detailed API reference to Apple's
Accelerate? I couldn't get any details for dsyev(r) by searching the web.
It doesn't matter much for gretl after your workaround solves the issue,
but in principle I guess it's not clear it is really a bug, perhaps they
just don't follow the netlib benchmark in this detail. But if it is,
perhaps we should report it.
In any case, excellent work of course!
thanks
Sven
5:59 p.m.
On Sun, 25 Oct 2020, Sven Schreiber wrote:
Am 25.10.2020 um 16:04 schrieb Allin Cottrell:
> On Sun, 25 Oct 2020, Allin Cottrell wrote:
>
>> So an interesting test on macOS would be to set GRETL_OLD_EV=1 and see
>> if the problem persists.
>
> I just tried on macOS 10.15.6: I got the crash that Fred reported, but
> when I set GRETL_OLD_EV=1 there was no crash.
>
> It turns out this is an Apple bug: there are two arguments to DSYEVR
> that are not supposed to be referenced when the function is called in
> the mode gretl uses (according to the netlib LAPACK docs). So we were
> just passing NULL for those arguments. But apparently Apple's DSYEVR
> does reference them, hence a crash on dereferencing a null pointer. In
> git (snapshots to follow) we now pass pointers to dummy arguments in
> those slots as a workaround.
Do you have any link to the detailed API reference to Apple's
Accelerate? I couldn't get any details for dsyev(r) by searching the web.
The canonical documentation for dsyevr should turn up by simply
googling "dsyevr", for example
http://www.netlib.org/lapack/explore-html/d2/d8a/group__double_s_yeigen_g...
http://www.netlib.org/lapack/lapack-3.1.1/html/dsyevr.f.html
It doesn't matter much for gretl after your workaround solves the
issue,
but in principle I guess it's not clear it is really a bug, perhaps they
just don't follow the netlib benchmark in this detail.
There may be no formally designated standard but I think most would
accept the netlib doc as the de facto standard.
I haven't found any formal documentation of the API for Apple's
lapack library (called by reference to the Accelerate Framework).
You may be right that they're not too bothered about adhering to the
netlib standard.
Once upon a time netlib lapack (version 3.2.1) had the same problem,
but that was fixed in the 3.2.2 release (June 2010) following a bug
report. See the comment on bug 0043 in
http://www.netlib.org/lapack/lapack-3.2.2.html
Allin
6:03 p.m.
Am 25.10.2020 um 18:59 schrieb Allin Cottrell:
On Sun, 25 Oct 2020, Sven Schreiber wrote:
>
> Do you have any link to the detailed API reference to Apple's
> Accelerate? I couldn't get any details for dsyev(r) by searching the web.
The canonical documentation for dsyevr should turn up by simply googling
"dsyevr", for example
Yep, I meant "their" dsyevr incarnation, the standard netlib stuff is
known of course.
Once upon a time netlib lapack (version 3.2.1) had the same problem,
but
that was fixed in the 3.2.2 release (June 2010) following a bug report.
See the comment on bug 0043 in
Ah OK, also interesting.
thanks
sven
8:42 p.m.
There was a recent mail discussion about R and MKL math library.
See https://stat.ethz.ch/pipermail/r-sig-mac/2020-September/013731.html
The last item in the discussion was from Jeroen Ooms and mentioned that both the Mac
package manager Homebrew and Python have removed all dependency on the Accelerate library
"... due to bugs and Apple's (lack of)
maintenance ..."
See this : https://github.com/scipy/scipy/wiki/Dropping-support-for-Accelerate
LAPACK in the Accelerate framework is supposed to be quite outdated.
As far as I know Apple has not provided source of its implementation.
I do know that the macOS implementation of R (provided by the R project) does not use the
Accelerate framework (because of bugs).
Berend Hasselman
On 25 Oct 2020, at 19:03, Sven Schreiber <svetosch(a)gmx.net>
wrote:
Am 25.10.2020 um 18:59 schrieb Allin Cottrell:
> On Sun, 25 Oct 2020, Sven Schreiber wrote:
>>
>> Do you have any link to the detailed API reference to Apple's
>> Accelerate? I couldn't get any details for dsyev(r) by searching the web.
>
> The canonical documentation for dsyevr should turn up by simply googling
> "dsyevr", for example
Yep, I meant "their" dsyevr incarnation, the standard netlib stuff is
known of course.
> Once upon a time netlib lapack (version 3.2.1) had the same problem, but
> that was fixed in the 3.2.2 release (June 2010) following a bug report.
> See the comment on bug 0043 in
Ah OK, also interesting.
thanks
sven
_______________________________________________
Gretl-users mailing list -- gretl-users(a)gretlml.univpm.it
To unsubscribe send an email to gretl-users-leave(a)gretlml.univpm.it
Website: https://gretlml.univpm.it/postorius/lists/gretl-users.gretlml.univpm.it/
2:08 p.m.
Am 25.10.2020 um 21:42 schrieb Berend Hasselman:
There was a recent mail discussion about R and MKL math library.
See https://stat.ethz.ch/pipermail/r-sig-mac/2020-September/013731.html
The last item in the discussion was from Jeroen Ooms and mentioned that both the Mac
package manager Homebrew and Python have removed all dependency on the Accelerate library
"... due to bugs and Apple's (lack of)
maintenance ..."
See this : https://github.com/scipy/scipy/wiki/Dropping-support-for-Accelerate
LAPACK in the Accelerate framework is supposed to be quite outdated.
That's also very interesting - the version mentioned there is exactly
the same that Allin mentioned as containing the bug that was fixed 10
years ago.
So maybe it's not an Apple bug, but the bug is that Accelerate basically
seems to be no longer maintained.
cheers
sven
1539
days inactive
1541
days old
9 comments
5 participants
participants (5)
-
Allin Cottrell -
Berend Hasselman -
Fred Engst -
Riccardo (Jack) Lucchetti -
Sven Schreiber