Checking network input processing by Python for a multi-threaded server
Markus Elfring
Markus.Elfring at web.de
Sat May 4 12:56:09 EDT 2019
>>> Server.shutdown() sets a flag that tells the main server to /stop
>>> accepting new requests/.
>>
>> Can it be that this method should perform a bit more resource management
>> (according to the selected configuration like “socketserver.ThreadingMixIn”)?
>>
> There isn't much more it can do
I see further software design possibilities.
> -- it has been a long standing axiom that killing threads is not recommended.
This data processing approach will trigger various development challenges.
>>> You may also need to code logic to ensure any handler threads have completed
>>
>> Can a class like “BaseServer” be responsible for the determination
>> if all extra started threads (or background processes) finished their work
>> as expected?
>
> You are asking for changes to the Python library for a use case
> that is not common.
I find this view questionable.
> Normally connections to a server are independent and do not share common data
Would you like to clarify corresponding application statistics any further?
> -- if there is anything in common,
Do you identify any more shared functionality?
> it is likely stored in a database management system
An use case evolved into my need to work also with an ordinary Python list
variable for a simple storage interface.
> which itself will provide locking for updates,
It is nice when you can reuse such software.
> and the connection handler will have to be coded to handle retries
> if multiple connections try to update the same records.
I am not concerned about this aspect for my test case.
> Servers aren't meant to be started and shutdown at a rapid rate
Their run times can vary considerably.
> (it's called "serve_forever" for a reason).
Will an other term become more appropriate?
> If the socketserver module doesn't provide what you need,
It took a while to understand the observed software behaviour better.
> you are free to copy socketserver.py to some other file (myserver.py?),
> and modify it to fit your needs.
Will it help to clarify any software extensions with corresponding maintainers?
> Maybe have the function that spawns handler threads append the thread ID
> to a list, have the function that cleans up a handler thread at the end
> send its ID via a Queue object,
This approach can be reasonable to some degree.
> and have the master periodically (probably the same loop that checks
> for shutdown), read the Queue, and remove the received ID from the list
> of active threads.
I imagine that there are nicer design options available for notifications
according to thread terminations.
> On shutdown, you loop reading the Queue and removing IDs from the list
> of active threads until the list is empty.
Will a condition variable (or a semaphore) be more helpful here?
>> How much do the programming interfaces from the available classes support
>> the determination that submitted tasks were completely finished?
>>
> Read the library reference manual and, for those modules with Python
> source, the source files (threading.py, socketserver.py, queue.py...)
>
> The simpler answer is that these modules DON'T...
I suggest to improve the software situation a bit more.
> It is your responsibility.
This view can be partly appropriate.
> socketserver threading model is that the main server loops waiting
> for connection requests, when it receives a request it creates
> a handler thread, and then it completely forgets about the thread
I find that this technical detail can be better documented.
> -- the thread is independent and completes the handling of the request.
Would you occasionally like to wait on the return value from such
data processing?
(Are these threads joinable?)
> If you need to ensure everything has finished before starting
> the next server instance, you will have to write the extensions
> to socketserver.
I might achieve something myself while it can be also nice to achieve
adjustments together with other developers.
>> Should mentioned system constraints be provided already by the Python
>> function (or class) library?
>
> Again, the model for socketserver, especially in threaded mode, is that
> requests being handled are completely independent. shutdown() merely stops
> the master from responding to new requests.
I find this understanding of the software situation also useful.
> In a more normal situation, .shutdown() would be called
> and then the entire program would call exit.
* Do you expect any more functionality here than an exit from a single thread?
* Does this wording include the abortion of threads which were left over?
> It is NOT normal for a program to create a server, shut it down,
> only to then repeat the sequence.
Will your understanding of such an use case grow, too?
Regards,
Markus
More information about the Python-list
mailing list