<div dir="ltr">Dear Mazen,<div>I don't hink that the proposals you outlined are clear enough at the moment, and would benefit from addressing the following points.</div><div><br></div><div>- the penalty method is the most interesting part of the proposal. I think a bit more expansion on this point would be useful. For example, how easy would it be to turn any constrained problem into an objective function that could be used with any unconstrained minimizer? Could that respecification work with existing interfaces for expressing constraints? Could the existing minimizer interface be left untouched, and function wrappers or classes written to convert those constrained systems into a function given to unmodified existing code?</div><div><br></div><div>- Vectorisation in optimize sounds like a nice idea, but may be very difficult to implement in practice. From your document it seems that you're mainly proposing to vectorise derivative calculation using finite differences. The finite difference routine `approx_derivative` in _numdiff would have to be rewritten to perform in a vectorised manner, across different dimensions (R^m -> R->n), with a vectorised objective function. How would the stack trace from the initial `minimize` call through to approx_derivative have to change? (this is definitely way more than a weeks work). If you had profiling to demonstrate that vectorisation would provide a significant performance gain that would be great.</div><div><br></div><div>- it's unclear what the normalisation (section 5) refers to? Is this an automated way of selecting the correct value for a Lagrangian multiplier?</div><div><br></div><div>- is the smart initialisation (section 7.2) a way of selecting the correct value for a Lagrangian multiplier used for the constrained->unconstrained transform? Or is it a variation on `brute + polish`? If the latter then how would this be any better than shgo/dual_annealing/differential_evolution?</div><div><br></div><div>- improvement of documentation is always welcomed, but there's not a huge explanation of the specific target areas for improvement.</div><div><br></div><div>regards,</div><div>Andrew.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 12 Apr 2021 at 19:06, Mazen Sayed <<a href="mailto:sayedmazen70@gmail.com">sayedmazen70@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hello,<div><br></div><div>I have changed the normalizer to be a user specified argument, also I have changed the initializer so the user can enter multiple data points, and the initializer will take the best initial point (that minimize the unconstrained problem), or the user can let the initializer guess the initial point by using random search, grid search or global optimizer.</div><div><br></div><div>Any modification before submission?</div><div><br></div><div>Thanks</div><div><br></div><div><a href="https://drive.google.com/file/d/1c9JOJgcq_Ss761rt9SOxnmEtuczuLzAb/view?usp=sharing" target="_blank">https://drive.google.com/file/d/1c9JOJgcq_Ss761rt9SOxnmEtuczuLzAb/view?usp=sharing</a> <br><br></div></div><br><img width="0" height="0" alt="" style="display: flex;" src="https://mailtrack.io/trace/mail/034ce5989f902345ca340fd7a18aaef540473ae7.png?u=6962871"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 11, 2021 at 7:26 PM Mazen Sayed <<a href="mailto:sayedmazen70@gmail.com" target="_blank">sayedmazen70@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br>This sounds really good and more reasonable.<div><br></div><div>Thanks for your help.<br><br><br></div></div><br><img width="0" height="0" alt="" style="display: flex;"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 11, 2021 at 7:08 PM Pamphile Roy <<a href="mailto:roy.pamphile@gmail.com" target="_blank">roy.pamphile@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><blockquote type="cite"><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto">I tend to agree with Daniel here, randomly choosing 50 points in a high-dimensional optimization space is not going to give any advantage. And why 50? </div></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto"><br></div><div dir="auto">The initialization part is one of the most important (and difficult to get right) part of any optimization algorithm, but this is mostly true for global ones: differential evolution, SHGO, Dual Annealing they’re all have their own way. Some of these and many others (especially local algorithms) rely on the user to explicitly pass an initial guess and take it from there.</div></div></blockquote><br></div><div>Agreed here, the random sampling must not be totally random in order to cover the parameter space in the most efficient way.</div><div>The global optimizers all use QMC methods (scipy.stats.qmc) so here you can just rely on these too.</div><div><br></div><div>Cheers,</div><div>Pamphile</div><br></div>_______________________________________________<br>
SciPy-Dev mailing list<br>
<a href="mailto:SciPy-Dev@python.org" target="_blank">SciPy-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scipy-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br>
</blockquote></div></div>
</blockquote></div></div>
_______________________________________________<br>
SciPy-Dev mailing list<br>
<a href="mailto:SciPy-Dev@python.org" target="_blank">SciPy-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scipy-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">_____________________________________<br>Dr. Andrew Nelson<br><br><br>_____________________________________</div>