<div><div dir="auto">Hi,</div><div><br><div class="gmail_quote"></div></div></div><div><div dir="ltr" class="gmail_attr">On Sun, 11 Apr 2021 at 15.09, Daniel Schmitz <<a href="mailto:danielschmitzsiegen@googlemail.com" target="_blank">danielschmitzsiegen@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="ltr"><div>Hey Sayed,</div><div><br></div><div>my two cents, not being a CoreDeveloper but a python developer interested in Optimization algorithms. <br></div><div><br></div><div>The automatic reformulation of the constrained problem into an unconstrained problem sounds similar to nlopt's augmented lagrangian: <a href="https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/#augmented-lagrangian-algorithm" target="_blank">https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/#augmented-lagrangian-algorithm</a> . I think this would be a great addition to scipy.optimize. I imagine that you would pass the reformulated objective to minimize then and just reuse the existing algorithms. <br></div><div><br></div><div>One objection to your idea about "smart initialization": why exactly 50 points and how exactly would they be sampled if no bounds are provided? Theoretically, a grid search over samples generated by for example latin hypercube sampling within a bounded volume could be a better initialization than a random guess. But I am not sure that this is in many cases a good idea. If you have no idea how to initialize your optimizer, I would go for one of the global optimizers.</div></div></blockquote><div dir="auto"><br></div><div dir="auto">My 2 cents too - I’m not a SciPy developer but I am passionate about optimization.</div><div dir="auto"><br></div></div><div><div dir="auto">I tend to agree with Daniel here, randomly choosing 50 points in a high-dimensional optimization space is not going to give any advantage. And why 50? </div><div dir="auto"><br></div><div dir="auto">The initialization part is one of the most important (and difficult to get right) part of any optimization algorithm, but this is mostly true for global ones: differential evolution, SHGO, Dual Annealing they’re all have their own way. Some of these and many others (especially local algorithms) rely on the user to explicitly pass an initial guess and take it from there.</div><div dir="auto"><br></div><div dir="auto">As for the penalty approach, I do agree it would be a nice addition: you may want to take a look - whether for inspiration or out of curiosity - at the Mystic library (<div dir="auto"><a href="https://github.com/uqfoundation/mystic">https://github.com/uqfoundation/mystic</a>). I believe the author has covered most use cases in terms of penalty/barrier methods. Speaking of penalty/barrier approaches, you may also wish to consider whether a constraint violation results into a point for which the objective function *cannot* be evaluated or simply a point you don’t want your algorithm to go (but the objective function can be evaluated there).</div><div dir="auto"><br></div><div dir="auto">As for the normalization/denormalization concept, I think it should definitely be a feature of all algorithms - and honestly I would expect anyone knowledgeable on optimization to always apply some sort of normalization if the parameters values spans multiple order of magnitude. There’s quite a few algorithms already that apply normalization internally no matter what, and of course this process helps in the vast majority of optimization problems (why would you make your algorithm sweat in handling different variables spanning 10 orders of magnitude?).</div><div dir="auto"><br></div><div dir="auto">Andrea.</div><div dir="auto"><br></div></div></div><div><div><div class="gmail_quote"><div dir="auto"><br></div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="ltr"><div dir="auto"></div><div><br></div><div>Best,</div><div><br></div><div>Daniel<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 11 Apr 2021 at 14:41, Mazen Sayed <<a href="mailto:sayedmazen70@gmail.com" target="_blank">sayedmazen70@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="ltr"><div>Dear,<div><br></div><div>I hope this email finds you well, this is my proposal for scipy.optimize project, I'm really interested to work on this project.</div><div><br></div><div>Thanks</div></div><div><br></div><a href="https://drive.google.com/file/d/12Q6NnorN74VkuQw_HRx2kuY-FIoK90V0/view?usp=sharing" target="_blank">https://drive.google.com/file/d/12Q6NnorN74VkuQw_HRx2kuY-FIoK90V0/view?usp=sharing</a><br><br><br><br><img alt="" style="display: flex;" src="https://mailtrack.io/trace/mail/65051114ea0a559072451a7b6dc3896d125b2103.png?u=6962871" width="0" height="0"></div><div id="m_-8203804488649032660m_-111851643062817127gmail-m_6919920764018287639DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br>
<table style="border-top-width:1px;border-top-style:solid;border-top-color:rgb(211,212,222)">
<tbody><tr>
<td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" style="width: 46px; height: 29px;" width="46" height="29"></a></td>
<td style="width:470px;padding-top:12px;font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px;color:rgb(65,66,78)">Virus-free. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link" style="font-family:Arial,Helvetica,sans-serif;color:rgb(68,83,234)" target="_blank">www.avast.com</a>
</td>
</tr>
</tbody></table><a href="#m_-8203804488649032660_m_-111851643062817127_m_6919920764018287639_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div>
_______________________________________________<br>
SciPy-Dev mailing list<br>
<a href="mailto:SciPy-Dev@python.org" target="_blank">SciPy-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scipy-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br>
</blockquote></div>
_______________________________________________<br>
SciPy-Dev mailing list<br>
<a href="mailto:SciPy-Dev@python.org" target="_blank">SciPy-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scipy-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br>
</blockquote></div></div>
</div>