Hi Nick, I have just checked all the links you posted, they are indeed very interesting and very efficient. However, I think those are very complicate in terms of installation and setup, and I still see a lot of usages for a multi-process scheduler. 2016-08-28 20:32 GMT-07:00 Nick Coghlan <ncoghlan@gmail.com>:
On 29 August 2016 at 11:50, Thales filizola costa <thalesfc@gmail.com> wrote:
What do you guys think? How to improve it? Is it relevant enough to be incorporated to std python ?
There are actually quite a few distributed schedulers out there (which can expand beyond a single machine), but "python multiprocess scheduler" isn't likely to bring them up in a web search (as when you're limited to a single machine, multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor is generally already good enough).
At a Python level, Celery is probably the most popular option for that: http://www.celeryproject.org/
Another well-established option is Kamaelia: http://www.kamaelia.org/Home. html
Dask is a more recent alternative specifically focused on computational tasks: http://dask.pydata.org/en/latest/
Once you move outside Python specific tooling, there are even more language independent options to play with, including the likes of Mesos and Kubernetes.
Cheers, Nick.
P.S. It's a fairly sad indictment of our industry that people think this is a sensible question to ask in developer interviews - the correct answer from a business efficiency perspective is "I wouldn't, I would use an existing open source task scheduler rather than inventing my own", just as the correct answer to "How would you implement a sort algorithm?" from that perspective is "I wouldn't, as the Python standard library's sorting implementation is vastly superior to anything I could come up with in 5 minutes on a whiteboard, and the native sorting capabilities of databases are also excellent". Reimplementing existing software from first principles is a great learning exercise, but it's not particularly relevant to the task of day-to-day software construction in most organisations.
(Alternatively, if the answer the interviewer is looking for is "I wouldn't, I would use...", then it may be an unfair "Gotcha!" question, and those aren't cool either, since they expect the interviewee to be able to read the interviewer's mind, rather than just answering the specific question they were asked)
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia