[SciPy-Dev] GSoC SciPy Optimization Ideas

[Andrea Gavana]

Hi,

On Sun, 11 Apr 2021 at 15.09, Daniel Schmitz <
danielschmitzsiegen at googlemail.com> wrote:

> Hey Sayed,
>
> my two cents, not being a CoreDeveloper but a python developer interested
> in Optimization algorithms.
>
> The automatic reformulation of the constrained problem into an
> unconstrained problem sounds similar to nlopt's augmented lagrangian:
> https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/#augmented-lagrangian-algorithm
> . I think this would be a great addition to scipy.optimize. I imagine that
> you would pass the reformulated objective to minimize then and just reuse
> the existing algorithms.
>
> One objection to your idea about "smart initialization": why exactly 50
> points and how exactly would they be sampled if no bounds are provided?
> Theoretically, a grid search over samples generated by for example latin
> hypercube sampling within a bounded volume could be a better initialization
> than a random guess. But I am not sure that this is in many cases a good
> idea. If you have no idea how to initialize your optimizer, I would go for
> one of the global optimizers.
>

My 2 cents too - I’m not a SciPy developer but I am passionate about
optimization.

I tend to agree with Daniel here, randomly choosing 50 points in a
high-dimensional optimization space is not going to give any advantage. And
why 50?

The initialization part is one of the most important (and difficult to get
right) part of any optimization algorithm, but this is mostly true for
global ones: differential evolution, SHGO, Dual Annealing they’re all have
their own way. Some of these and many others (especially local algorithms)
rely on the user to explicitly pass an initial guess and take it from there.

As for the penalty approach, I do agree it would be a nice addition: you
may want to take a look - whether for inspiration or out of curiosity - at
the Mystic library (
https://github.com/uqfoundation/mystic). I believe the author has covered
most use cases in terms of penalty/barrier methods. Speaking of
penalty/barrier approaches, you may also wish to consider whether a
constraint violation results into a point for which the objective function
*cannot* be evaluated or simply a point you don’t want your algorithm to go
(but the objective function can be evaluated there).

As for the normalization/denormalization concept, I think it should
definitely be a feature of all algorithms - and honestly I would expect
anyone knowledgeable on optimization to always apply some sort of
normalization if the parameters values spans multiple order of magnitude.
There’s quite a few algorithms already that apply normalization internally
no matter what, and of course this process helps in the vast majority of
optimization problems (why would you make your algorithm sweat in handling
different variables spanning 10 orders of magnitude?).

Andrea.




> Best,
>
> Daniel
>
> On Sun, 11 Apr 2021 at 14:41, Mazen Sayed <sayedmazen70 at gmail.com> wrote:
>
>> Dear,
>>
>> I hope this email finds you well, this is my proposal for scipy.optimize
>> project, I'm really interested to work on this project.
>>
>> Thanks
>>
>>
>> https://drive.google.com/file/d/12Q6NnorN74VkuQw_HRx2kuY-FIoK90V0/view?usp=sharing
>>
>>
>>
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
>> www.avast.com
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>> <#m_-8203804488649032660_m_-111851643062817127_m_6919920764018287639_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20210411/945e6fb5/attachment.html>