<div dir="auto">For the random initialization I would propose to write a dedicated new function called for example randomsearch (familiar name as in sklearn) as a counterpart to brute_force. Given some bounds, a function and a number of samples, it should return the best sample. <div dir="auto"><br></div><div dir="auto">Adding randomsearch as a method for x0 in minimize would then accomplish what you wanted originally. Not sure though what the core developers think about this from an API perspective.</div><div dir="auto"><br></div><div dir="auto">Cheers</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Mazen Sayed <<a href="mailto:sayedmazen70@gmail.com">sayedmazen70@gmail.com</a>> schrieb am So., 11. Apr. 2021, 16:33:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">Dear,<div><br></div><div>For the first point, 50 was an example, the number of points is a hyperparameter that the user should define first, but I agree with you that it may be useless specially with higher dimensions.</div><div>For the second point, Many thanks for the resources, I will read them carefully.</div><div>For the third point, I agree with you that any one who studied optimization, should normalize first, I was suggesting that we can add this normalization feature, and the user can specify a normalize parameter to do this step for him.</div><div><br></div><div>Thanks,<br></div><div>Mazen</div><div><br></div></div><br><img width="0" height="0" alt="" style="display:flex" src="https://mailtrack.io/trace/mail/0bc6784d4189adf9ed769c90a3710b3ac9866601.png?u=6962871"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 11, 2021 at 4:20 PM Andrea Gavana <<a href="mailto:andrea.gavana@gmail.com" target="_blank" rel="noreferrer">andrea.gavana@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="auto">Hi,</div><div><br><div class="gmail_quote"></div></div></div><div><div dir="ltr" class="gmail_attr">On Sun, 11 Apr 2021 at 15.09, Daniel Schmitz <<a href="mailto:danielschmitzsiegen@googlemail.com" target="_blank" rel="noreferrer">danielschmitzsiegen@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hey Sayed,</div><div><br></div><div>my two cents, not being a CoreDeveloper but a python developer interested in Optimization algorithms. <br></div><div><br></div><div>The automatic reformulation of the constrained problem into an unconstrained problem sounds similar to nlopt's augmented lagrangian: <a href="https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/#augmented-lagrangian-algorithm" target="_blank" rel="noreferrer">https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/#augmented-lagrangian-algorithm</a> . I think this would be a great addition to scipy.optimize. I imagine that you would pass the reformulated objective to minimize then and just reuse the existing algorithms. <br></div><div><br></div><div>One objection to your idea about "smart initialization": why exactly 50 points and how exactly would they be sampled if no bounds are provided? Theoretically, a grid search over samples generated by for example latin hypercube sampling within a bounded volume could be a better initialization than a random guess. But I am not sure that this is in many cases a good idea. If you have no idea how to initialize your optimizer, I would go for one of the global optimizers.</div></div></blockquote><div dir="auto"><br></div><div dir="auto">My 2 cents too - I’m not a SciPy developer but I am passionate about optimization.</div><div dir="auto"><br></div></div><div><div dir="auto">I tend to agree with Daniel here, randomly choosing 50 points in a high-dimensional optimization space is not going to give any advantage. And why 50? </div><div dir="auto"><br></div><div dir="auto">The initialization part is one of the most important (and difficult to get right) part of any optimization algorithm, but this is mostly true for global ones: differential evolution, SHGO, Dual Annealing they’re all have their own way. Some of these and many others (especially local algorithms) rely on the user to explicitly pass an initial guess and take it from there.</div><div dir="auto"><br></div><div dir="auto">As for the penalty approach, I do agree it would be a nice addition: you may want to take a look - whether for inspiration or out of curiosity - at the Mystic library (<div dir="auto"><a href="https://github.com/uqfoundation/mystic" target="_blank" rel="noreferrer">https://github.com/uqfoundation/mystic</a>). I believe the author has covered most use cases in terms of penalty/barrier methods. Speaking of penalty/barrier approaches, you may also wish to consider whether a constraint violation results into a point for which the objective function *cannot* be evaluated or simply a point you don’t want your algorithm to go (but the objective function can be evaluated there).</div><div dir="auto"><br></div><div dir="auto">As for the normalization/denormalization concept, I think it should definitely be a feature of all algorithms - and honestly I would expect anyone knowledgeable on optimization to always apply some sort of normalization if the parameters values spans multiple order of magnitude. There’s quite a few algorithms already that apply normalization internally no matter what, and of course this process helps in the vast majority of optimization problems (why would you make your algorithm sweat in handling different variables spanning 10 orders of magnitude?).</div><div dir="auto"><br></div><div dir="auto">Andrea.</div><div dir="auto"><br></div></div></div><div><div><div class="gmail_quote"><div dir="auto"><br></div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="auto"></div><div><br></div><div>Best,</div><div><br></div><div>Daniel<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 11 Apr 2021 at 14:41, Mazen Sayed <<a href="mailto:sayedmazen70@gmail.com" target="_blank" rel="noreferrer">sayedmazen70@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Dear,<div><br></div><div>I hope this email finds you well, this is my proposal for scipy.optimize project, I'm really interested to work on this project.</div><div><br></div><div>Thanks</div></div><div><br></div><a href="https://drive.google.com/file/d/12Q6NnorN74VkuQw_HRx2kuY-FIoK90V0/view?usp=sharing" target="_blank" rel="noreferrer">https://drive.google.com/file/d/12Q6NnorN74VkuQw_HRx2kuY-FIoK90V0/view?usp=sharing</a><br><br><br><br><img alt="" style="display:flex" width="0" height="0"></div><div id="m_421618496721677197gmail-m_6208003926892838107m_-8203804488649032660m_-111851643062817127gmail-m_6919920764018287639DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br>
<table style="border-top:1px solid rgb(211,212,222)">
<tbody><tr>
<td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon" target="_blank" rel="noreferrer"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" style="width:46px;height:29px" width="46" height="29"></a></td>
<td style="width:470px;padding-top:12px;font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px;color:rgb(65,66,78)">Virus-free. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link" style="font-family:Arial,Helvetica,sans-serif;color:rgb(68,83,234)" target="_blank" rel="noreferrer">www.avast.com</a>
</td>
</tr>
</tbody></table><a href="#m_421618496721677197_m_6208003926892838107_m_-8203804488649032660_m_-111851643062817127_m_6919920764018287639_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1" rel="noreferrer"></a></div>
_______________________________________________<br>
SciPy-Dev mailing list<br>
<a href="mailto:SciPy-Dev@python.org" target="_blank" rel="noreferrer">SciPy-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scipy-dev" rel="noreferrer noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br>
</blockquote></div>
_______________________________________________<br>
SciPy-Dev mailing list<br>
<a href="mailto:SciPy-Dev@python.org" target="_blank" rel="noreferrer">SciPy-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scipy-dev" rel="noreferrer noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br>
</blockquote></div></div>
</div>
_______________________________________________<br>
SciPy-Dev mailing list<br>
<a href="mailto:SciPy-Dev@python.org" target="_blank" rel="noreferrer">SciPy-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scipy-dev" rel="noreferrer noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br>
</blockquote></div></div>
_______________________________________________<br>
SciPy-Dev mailing list<br>
<a href="mailto:SciPy-Dev@python.org" target="_blank" rel="noreferrer">SciPy-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scipy-dev" rel="noreferrer noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br>
</blockquote></div>