Re: bfgsmin and fminunc

[Jaroslav Hajek]

On Thu, May 14, 2009 at 11:26 AM, Michael Creel <address@hidden> wrote:
>
> Oops, sorry. bfgsmin was using the analytic gradient, and fminunc was using
> numeric. If you edit the script to set the objective function to objective2,
> then both will use numeric gradients. The speed difference becomes less than
> 2X, but the convergence tolerance issue remains.
> M.
>

Hi Michael,

I'm glad to find you took the time to try some testing. The Octave's
new fminunc got very little testing so far, so I won't be surprised at
all if major bugs, robustness defects or other problems are
discovered. fminunc is intended to be Matlab-compatible and
essentially mimicks the algorithm in fsolve (which was much more
tested). The speed of a single iteration depends crucially on whether
you have Octave linked with libqrupdate (if not, it will be O(N^3)).
And of course the number of iterations is a major issue.
Regarding the tolerances issue: the default values for both TolFun and
TolX are sqrt(eps(class(x0))), which is about
1.5e-8 in the double case (fminunc will work in single precision if
given single prec inputs).
in the convergence tests (as well as in TR stepping), fminunc uses
automatic scaling by the predicted second derivatives (hessian
diagonal) to achieve scaling independency. So this does not translate
to absolute bounds.
It would be nice to provide a way to specify absolute tolerances, but
I'm not sure how to best do it w.r.t. Matlab compatibility.

Unfortunately, I can't really try the comparison because I have
trouble to get bfgsmin working at all using development octave, there
are some problems with feval that seem to be nonetheless on Octave's
side.
I'll investigate what's wrong.

Using analytic gradient, fminunc seems to produce a much better result
(try if you like), so maybe part of the problem is the numeric
gradient calculation. The big difference I see is that unlike fminunc,
bfgsmin uses centered differences, which are likely more accurate. I
also see it uses a slightly more sophisticated delta selection, so I
may copy that to __fdjac__, too. Allowing (optionally) central
differences within both fsolve and fminunc would indeed be a good
idea.

Still, even with the analytic gradient I'm not particularly pleased
with fminunc's performance (number of iterations), so I'll look for
possible improvements.

Anyway, further tests/suggestions from your part are welcome.

regards

-- 
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz