[Gretl-devel] More about the test for analytical derivatives

As a result of some testing of my panel stochastic frontier function, I would like to give a warning about the reliability of the test for the accuracy of analytical derivatives. In general, it is very useful. But the warning is that I have found that I can provoke failures in the test simply by changing the starting values of my parameters. I am 99.9% sure that my analytical derivatives are now correct (though they are horribly tedious to program) because (a) I can reproduce results generated by Stata to the 6th significant figure using both numerical and analytical derivatives, and (b) Stata reports identical gradients for the parameters at the same starting values. From past experience I know that the log likelihood function for complex stochastic frontiers is not globally concave and getting good starting values is sometimes very difficult. Part of the reason is that the log likelihood involves an evaluation of the cumulative normal at what may be extreme values, which can cause degeneracy and arbitrary fixes. Stata frequently reports that it is in a non-concave part of the likelihood function. What I think happens in gretl is that the minpack numerical derivatives routine generates arc values for the derivatives that differ from the analytical derivatives because the slope changes abruptly. As a consequence the failure message is not necessarily correct or helpful, because the difference may be due to discontinuities or lack of concavity in the log-likelihood. There is a possible solution, but there may be restrictions on implementing it. Rather than stopping when the numerical and analytical derivatives appear to differ, why not continue but by using the numerical derivatives - reporting that fact. Then, every 5 or 10 iterations test whether they still differ. If the derivatives appear to differ, continue with the numerical derivatives whereas if they are now identical the program should switch to use the analytical derivatives. Gordon