Re: [Twisted-Python] SMB server component for twisted
On May 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega <wsanchez@wsanchez.net(mailto:wsanchez@wsanchez.net)> wrote: On May 7, 2020, at 12:48 AM, Glyph <glyph@twistedmatrix.com(mailto:glyph@twistedmatrix.com)> wrote:
If you want to include it in Twisted itself, your best bet is to actually develop it within twisted, as a series of small contributions, rather than as one gigantic one-shot one. Contributions over, say, 400 lines, take exponentially longer to review.
Developing it within Twisted will make things go slower; you'll need to get everything code reviewed, you'll need to support multiple versions of Python (no py2 any more, but py3.5 is still pretty old), you'll have to have full test coverage from the get-go. But doing these things from the start is much easier than trying to retrofit them.
I actually think that this would be a pretty good fit for Twisted, in the same way that it's been a benefit to have Conch maintained alongside the rest of Twisted. I can see you're developing things very much in line with Twisted's architecture (using cred for authentication, a realm interface, etc) and you've voiced this interest, so it would be great to have you along!
I think it's great to get an SMB implementation in the Twisted org, but why would we even consider adding something like this to the main Twisted project?
Just practically speaking, we've tried both approaches, and (very broadly generalizing) this one works and the other one doesn't. Separate projects that have been adopted from outside after they gained some traction are doing okay (i.e. Treq and Klein, some of the most popular applications for Twisted). But others (txacme) are suffering from a lack of maintenance and still others (ldaptor) just have a largely non-overlapping community and hardly benefit at all from org membership. And as you've noted, even the ones doing their best often want for reviews for far longer than Twisted's own latency.
The main repo is large, and including Twisted in your project brings along far more functionality than you are likely to use, and packages are a fine way to get the functionality you want, so what's the logic for including a new thing in Twisted proper?
Abstractly you make a plausible case. Concretely we tried to start splitting things up on the axes of protocol support and it just made a giant mess and created lots of overhead for the project. Modern infrastructure makes this slightly easier (last time we tried to start this process we didn't have a Github or a Travis), but not much easier. And even modern, better infrastructure leads to an avalanche of tedium as we copy CI configuration back and forth endlessly, replicating lessons that we learn about travis, and azure, and codecov, and (ad nauseam) Even in the abstract case though there’s a question of what twisted “is” and on what axes it should be split up. I think splitting up by protocol was largely a mistake, but splitting some of the really low level core stuff out (towncrier, automat, and hopefully filepath and deferred) has been pretty successful and worthwhile. Particularly in a post-asyncio world, the core event loop isn’t super interesting and the broad protocol support is the appeal. So even in the abstract case, there’s a case to be made that twisted is a bunch of implementations and interfaces for wire protocols that involve navigating hierarchies of things (DNS, HTTP, IMAP, SFTP) and SMB fits right in there.
You cite Conch above, where I'd actually argue that Conch should be moved to a separate repo. Am I crazy?
Not crazy, there's a valid case to be made, but Conch has received a tremendous amount of routine maintenance keeping it up to date with Twisted and Python API evolution by members of the project who would almost certainly otherwise not bother, just to keep the test suite going and the project as a whole integrated. This allows it to remain viable for the much smaller number of SSH-specific contributions that come along.
That said, a counter argument is that projects in the Twisted org suffer greatly from poor participation and strict development rules, resulting in glacial progress. It's not unusual for me to have a PR take 6 months to a year to get a review (I have two right now that have been waiting since November), and perhaps such PRs would get some attention in the main repo. But I think that's an unfortunate way to address that problem.
This is just a natural consequence of affordances provided by the tooling we presently have. Personally I look at the global queue of pull requests across all these projects and it's kind of exhausting. I don't think many other people do. Case in point: how many ldaptor or txacme reviews have you done since November? This is not to throw any shade; I’m not saying you should have, but it illustrates that right now, separate repos engender a really unfortunate level of community fragmentation. I agree that this is not really an ideal situation, but addressing it requires building a degree of tooling that we just don’t have the resources to do. Our previous attempt to do so was … bad; it kind of illustrated that we don’t have the relevant expertise to build these kinds of tools and mores either, and we just got kind of lucky with the somewhat random pile of jank we have landed on with the current twistedmatrix.com process. Anyway if someone wanted to do the work to separate out the concerns of version control, pip installability, issue tracking, and integrated regression testing for a whole suite of related python packages in an org, I would be extremely excited, but the benefits of just having Twisted be, say, 5% bigger overall to foster development of a whole new popular and intimidating protocol seem clear given the state of the world today. If those tools do show up later to make this tractable, we can easily split it out later! Along with conch, and mail, and names, for that matter… -glyph
On 15/05/2020 10:52 am, Glyph wrote:
y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega <wsanchez@wsanche <mailto:wsanchez@wsanchez.net>
I think it's great to get an SMB implementation in the Twisted org, but why would we even consider adding something like this to the main Twisted project?
The advantage of twisted itself is cross-protocol abstractions such as cred. Of course you don't have to be in the repo to use them, but in practice developers need the discipline of a single project to maintain consistency, otherwise the human tendency to reinvent wheels is too strong. Ian
On 16/05/2020 10:55 am, Ian Haywood wrote:
On 15/05/2020 10:52 am, Glyph wrote:
y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega <wsanchez@wsanche <mailto:wsanchez@wsanchez.net>
I think it's great to get an SMB implementation in the Twisted org, but why would we even consider adding something like this to the main Twisted project?
The advantage of twisted itself is cross-protocol abstractions such as cred. Of course you don't have to be in the repo to use them, but in practice developers need the discipline of a single project to maintain consistency, otherwise the human tendency to reinvent wheels is too strong
Apropos we have two APIs for exporting filesystems which are broadly similar: SFTP in conch and FTP itself, SMB is about to be a third, ideally we should have one or have them descend from one another. It's slightly harder than it sounds as SMB is a file-access, as opposed to file-transfer, protocol and supports some extra features such as locking. Ian
On May 15, 2020, at 6:14 PM, Ian Haywood <ian@haywood.id.au> wrote:
On 16/05/2020 10:55 am, Ian Haywood wrote:
On 15/05/2020 10:52 am, Glyph wrote:
y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega <wsanchez@wsanche <mailto:wsanchez@wsanchez.net>
I think it's great to get an SMB implementation in the Twisted org, but why would we even consider adding something like this to the main Twisted project?
The advantage of twisted itself is cross-protocol abstractions such as cred. Of course you don't have to be in the repo to use them, but in practice developers need the discipline of a single project to maintain consistency, otherwise the human tendency to reinvent wheels is too strong
Apropos we have two APIs for exporting filesystems which are broadly similar: SFTP in conch and FTP itself, SMB is about to be a third, ideally we should have one or have them descend from one another. It's slightly harder than it sounds as SMB is a file-access, as opposed to file-transfer, protocol and supports some extra features such as locking.
This does point out one of my secret hopes for SMB: that a file server's maintainers will care enough about file throughput that we'll finally get a centralized, official way of doing async file I/O that we can share with SFTP, FTP, and HTTP :). -glyph
On 16/05/2020 06:55, Glyph wrote:
This does point out one of my secret hopes for SMB: that a file server's maintainers will care enough about file throughput that we'll finally get a centralized, official way of doing async file I/O that we can share with SFTP, FTP, and HTTP :).
I got excited by the title, but this all seems to be able giving non-filesystems a file-system like API. If I've missed something, please let me know, but I couldn't find an async interface for writing to actual file systems. It feels like Twisted has just punted on them as "fast enough" to not need async interfaces, but that can really not be the case for networked filesystems (NFS, GPFS, etc) where the blocking time for reading or writing can be seconds (or minutes if it's having a bad day!). What's the Twisted solution for these kinds of things? Defer all file IO into a thread? Chris
On May 15, 2020, at 11:30 PM, Chris Withers <chris@withers.org> wrote:
On 16/05/2020 06:55, Glyph wrote:
This does point out one of my secret hopes for SMB: that a file server's maintainers will care enough about file throughput that we'll finally get a centralized, official way of doing async file I/O that we can share with SFTP, FTP, and HTTP :).
I got excited by the title, but this all seems to be able giving non-filesystems a file-system like API.
What are you referring to as non-filesystems?
If I've missed something, please let me know, but I couldn't find an async interface for writing to actual file systems.
In Twisted? There isn't one, really, which is exactly the issue. There are various operating system interfaces for this but none of them are great.
It feels like Twisted has just punted on them as "fast enough" to not need async interfaces, but that can really not be the case for networked filesystems (NFS, GPFS, etc) where the blocking time for reading or writing can be seconds (or minutes if it's having a bad day!).
Indeed, I'm well aware! That's what I'm talking about. It's not quite worth the energy to do it just for your average HTTP server, but it sure would be nice if we did have it, since it works so much better in the edge cases you enumerated. (Or even against a spinning rust platter under significant load.)
What's the Twisted solution for these kinds of things? Defer all file IO into a thread?
That is the only really portable solution, but there are sometimes platform-specific ones which can do better in some circumstances. However, if we had a unified API for accessing the filesystem that presented a non-blocking interface without *talking* about threads, we'd be able to take user code written against that API and instead make it do something smarter than just using a thread - or even just use an intelligent number of threads (one per platter, for example, which you can figure out by looking at various files in /etc, on some platforms, sometimes). -glyph
On 16/05/2020 07:42, Glyph wrote:
On May 15, 2020, at 11:30 PM, Chris Withers <chris@withers.org <mailto:chris@withers.org>> wrote:
On 16/05/2020 06:55, Glyph wrote:
This does point out one of my secret hopes for SMB: that a file server's maintainers will care enough about file throughput that we'll finally get a centralized, official way of doing async file I/O that we can share with SFTP, FTP, and HTTP :).
I got excited by the title, but this all seems to be able giving non-filesystems a file-system like API.
What are you referring to as non-filesystems?
Ah, sorry, misread the above.
If I've missed something, please let me know, but I couldn't find an async interface for writing to actual file systems.
In Twisted? There isn't one, really, which is exactly the issue. There are various operating system interfaces for this but none of them are great.
What's the asyncio spin on this? We server like uvicorn are all writing logs, are they too just punting on the blocking nature of writing to filesystems? Chris
On Saturday, 16 May 2020 02:14:57 BST Ian Haywood wrote:
On 16/05/2020 10:55 am, Ian Haywood wrote:
On 15/05/2020 10:52 am, Glyph wrote:
y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega <wsanchez@wsanche <mailto:wsanchez@wsanchez.net>
I think it's great to get an SMB implementation in the Twisted org, but why would we even consider adding something like this to the main Twisted project?
The advantage of twisted itself is cross-protocol abstractions such as cred. Of course you don't have to be in the repo to use them, but in practice developers need the discipline of a single project to maintain consistency, otherwise the human tendency to reinvent wheels is too strong
Apropos we have two APIs for exporting filesystems which are broadly similar: SFTP in conch and FTP itself, SMB is about to be a third, ideally we should have one or have them descend from one another. It's slightly harder than it sounds as SMB is a file-access, as opposed to file-transfer, protocol and supports some extra features such as locking.
I do not see a common class tree works for things that are so different as SMB and FTP. Barry
Ian
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Is anyone going to mention twisted.vfs? On Tue, May 19, 2020 at 8:33 AM Barry Scott <barry.scott@forcepoint.com> wrote:
On Saturday, 16 May 2020 02:14:57 BST Ian Haywood wrote:
On 16/05/2020 10:55 am, Ian Haywood wrote:
On 15/05/2020 10:52 am, Glyph wrote:
y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega <wsanchez@wsanche <mailto:wsanchez@wsanchez.net>
I think it's great to get an SMB implementation in the Twisted org, but why would we even consider adding something like this to the main Twisted project?
The advantage of twisted itself is cross-protocol abstractions such as cred. Of course you don't have to be in the repo to use them, but in practice developers need the discipline of a single project to maintain consistency, otherwise the human tendency to reinvent wheels is too strong
Apropos we have two APIs for exporting filesystems which are broadly similar: SFTP in conch and FTP itself, SMB is about to be a third, ideally we should have one or have them descend from one another. It's slightly harder than it sounds as SMB is a file-access, as opposed to file-transfer, protocol and supports some extra features such as locking.
I do not see a common class tree works for things that are so different as SMB and FTP.
Barry
Ian
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On May 19, 2020, at 5:40 AM, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
Is anyone going to mention twisted.vfs?
You, apparently? :) If anyone is curious about this code, `git revert 2059b5313ec236e8c3f77da4bd773ab98299ca1b` to have a look at it. twisted.vfs was an attempt to do something like this, but it also suffered from the split-release infrastructure; it wasn't released so it didn't get user feedback, its interfaces were odd in several ways, it was very specific to POSIX filesystems, had some ambiguity around whether it wanted to be operating in terms of strings or objects, and generally didn't provide much value. If we want to bring this concept back (as https://twistedmatrix.com/trac/ticket/4934 <https://twistedmatrix.com/trac/ticket/4934> alludes to by mentioning "twisted.path") it would benefit from lots of discussion and a concerted effort to integrate into the existing HTTP/SFTP/FTP code, so we actually create an interface that makes it easy for applications to create virtual filesystems accessible in multiple ways, with non-filesystem backends, rather than just wrapping up our file I/O code in a translation layer between... our versions of POSIX constants and the 'os' module's version of those same constants. Nevertheless 'vfs' may be of archaeological interest for someone embarking on such a project; it's not all bad ideas, it just had some problems :). -g
Hello, I am developing a twisted app, and it runs as a service using twistd -y to start the app. I am having a problem of running a thread. I know it is not recommended to use thread, but the library that I use, the object created is running in a thread. Here is the problem: 1) if I start reactor by running reactor.run() directly, thread is running fine 2) if I run it as a service using twisted, the thread is not running, it runs but it seems to be blocked, because I tried to write something to file using time.sleep(), but file is empty. Is there something that I miss? How can I debug this? Thank you, TS
On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH < touch.sereysethy@gmail.com> wrote:
Hello,
I am developing a twisted app, and it runs as a service using twistd -y to start the app. I am having a problem of running a thread. I know it is not recommended to use thread, but the library that I use, the object created is running in a thread.
Here is the problem:
1) if I start reactor by running reactor.run() directly, thread is running fine 2) if I run it as a service using twisted, the thread is not running, it runs but it seems to be blocked, because I tried to write something to file using time.sleep(), but file is empty.
Is there something that I miss? How can I debug this?
A good next step would be to create at Short, Self Contained, Correct (Compilable), Example <http://sscce.org/> and share it. Jean-Paul
Thank you,
TS _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi Jean-Paul, I found the error after working on a short, self contained, correcte example. It is not about twisted and normal thread in general, but something else. After more tests, I come to realise that whenever I instantiate this class PolicyClient from this library https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in my protocol or just anywhere, twisted app either freezes or terminates, and this *only* happens when the app runs as a daemon, created using a service (.tac file). I hope you can take a look at the class PolicyClient, why it causes problem to reactor main loop. Please point me to where the idea might come from. I can give you the example but not sure if it is enough, as the client needs to connect to a server. Thank you, TS On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH < touch.sereysethy@gmail.com> wrote:
Hello,
I am developing a twisted app, and it runs as a service using twistd -y to start the app. I am having a problem of running a thread. I know it is not recommended to use thread, but the library that I use, the object created is running in a thread.
Here is the problem:
1) if I start reactor by running reactor.run() directly, thread is running fine 2) if I run it as a service using twisted, the thread is not running, it runs but it seems to be blocked, because I tried to write something to file using time.sleep(), but file is empty.
Is there something that I miss? How can I debug this?
A good next step would be to create at Short, Self Contained, Correct (Compilable), Example <http://sscce.org/> and share it.
Jean-Paul
Thank you,
TS _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH <touch.sereysethy@gmail.com> wrote:
Hi Jean-Paul,
I found the error after working on a short, self contained, correcte example. It is not about twisted and normal thread in general, but something else.
After more tests, I come to realise that whenever I instantiate this class PolicyClient from this library https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in my protocol or just anywhere, twisted app either freezes or terminates, and this *only* happens when the app runs as a daemon, created using a service (.tac file).
I hope you can take a look at the class PolicyClient, why it causes problem to reactor main loop. Please point me to where the idea might come from.
I can give you the example but not sure if it is enough, as the client needs to connect to a server.
It looks like PolicyClient does a blocking HTTP request in __init__. Since PolicyClient is instantiated in the reactor thread, this will block the reactor. Perhaps this call is hanging for some reason when the process has daemonized? This would be unusual but not unheard of. For example, there are macOS environments where a process does not have access to the network if it is not associated with a windowing session. What platform do you observe the problem on, and what more can you learn about what the process does on its way to hanging (eg can you run strace on it)? Jean-Paul
Thank you, TS
On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH < touch.sereysethy@gmail.com> wrote:
Hello,
I am developing a twisted app, and it runs as a service using twistd -y to start the app. I am having a problem of running a thread. I know it is not recommended to use thread, but the library that I use, the object created is running in a thread.
Here is the problem:
1) if I start reactor by running reactor.run() directly, thread is running fine 2) if I run it as a service using twisted, the thread is not running, it runs but it seems to be blocked, because I tried to write something to file using time.sleep(), but file is empty.
Is there something that I miss? How can I debug this?
A good next step would be to create at Short, Self Contained, Correct (Compilable), Example <http://sscce.org/> and share it.
Jean-Paul
Thank you,
TS _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
I am on MacOS, it is a development phase, but deployment will be on Linux. It is hard to run dtrace/dtruss on MacOS due to the "system integrity protection" on MacOS. I got some output but very limited information. So if there is a workaround you might suggest? TS On Thu, May 21, 2020 at 10:20 PM Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH < touch.sereysethy@gmail.com> wrote:
Hi Jean-Paul,
I found the error after working on a short, self contained, correcte example. It is not about twisted and normal thread in general, but something else.
After more tests, I come to realise that whenever I instantiate this class PolicyClient from this library https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in my protocol or just anywhere, twisted app either freezes or terminates, and this *only* happens when the app runs as a daemon, created using a service (.tac file).
I hope you can take a look at the class PolicyClient, why it causes problem to reactor main loop. Please point me to where the idea might come from.
I can give you the example but not sure if it is enough, as the client needs to connect to a server.
It looks like PolicyClient does a blocking HTTP request in __init__. Since PolicyClient is instantiated in the reactor thread, this will block the reactor. Perhaps this call is hanging for some reason when the process has daemonized? This would be unusual but not unheard of. For example, there are macOS environments where a process does not have access to the network if it is not associated with a windowing session.
What platform do you observe the problem on, and what more can you learn about what the process does on its way to hanging (eg can you run strace on it)?
Jean-Paul
Thank you, TS
On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH < touch.sereysethy@gmail.com> wrote:
Hello,
I am developing a twisted app, and it runs as a service using twistd -y to start the app. I am having a problem of running a thread. I know it is not recommended to use thread, but the library that I use, the object created is running in a thread.
Here is the problem:
1) if I start reactor by running reactor.run() directly, thread is running fine 2) if I run it as a service using twisted, the thread is not running, it runs but it seems to be blocked, because I tried to write something to file using time.sleep(), but file is empty.
Is there something that I miss? How can I debug this?
A good next step would be to create at Short, Self Contained, Correct (Compilable), Example <http://sscce.org/> and share it.
Jean-Paul
Thank you,
TS _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, May 21, 2020 at 4:46 PM Sereysethy TOUCH <touch.sereysethy@gmail.com> wrote:
I am on MacOS, it is a development phase, but deployment will be on Linux.
It is hard to run dtrace/dtruss on MacOS due to the "system integrity protection" on MacOS. I got some output but very limited information.
So if there is a workaround you might suggest?
I would definitely recommend nailing down the exact cause of the problem. I think that system integrity protection can be bypassed, perhaps by creating a virtualenv? I am not particularly familiar with macOS but I'm sure there's some good information somewhere out there. If the problem is the blocking network operations performed by `PolicyClient` then you'll probably have to move even the instantiation of that class to a separate thread. This is probably good anyway, since it's not necessarily the case that it is safe to create the object in one thread and then use it in another. Better to create and use it all in the same thread. Jean-Paul
TS
On Thu, May 21, 2020 at 10:20 PM Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH < touch.sereysethy@gmail.com> wrote:
Hi Jean-Paul,
I found the error after working on a short, self contained, correcte example. It is not about twisted and normal thread in general, but something else.
After more tests, I come to realise that whenever I instantiate this class PolicyClient from this library https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in my protocol or just anywhere, twisted app either freezes or terminates, and this *only* happens when the app runs as a daemon, created using a service (.tac file).
I hope you can take a look at the class PolicyClient, why it causes problem to reactor main loop. Please point me to where the idea might come from.
I can give you the example but not sure if it is enough, as the client needs to connect to a server.
It looks like PolicyClient does a blocking HTTP request in __init__. Since PolicyClient is instantiated in the reactor thread, this will block the reactor. Perhaps this call is hanging for some reason when the process has daemonized? This would be unusual but not unheard of. For example, there are macOS environments where a process does not have access to the network if it is not associated with a windowing session.
What platform do you observe the problem on, and what more can you learn about what the process does on its way to hanging (eg can you run strace on it)?
Jean-Paul
Thank you, TS
On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH < touch.sereysethy@gmail.com> wrote:
Hello,
I am developing a twisted app, and it runs as a service using twistd -y to start the app. I am having a problem of running a thread. I know it is not recommended to use thread, but the library that I use, the object created is running in a thread.
Here is the problem:
1) if I start reactor by running reactor.run() directly, thread is running fine 2) if I run it as a service using twisted, the thread is not running, it runs but it seems to be blocked, because I tried to write something to file using time.sleep(), but file is empty.
Is there something that I miss? How can I debug this?
A good next step would be to create at Short, Self Contained, Correct (Compilable), Example <http://sscce.org/> and share it.
Jean-Paul
Thank you,
TS _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thursday, 21 May 2020 21:45:44 BST Sereysethy TOUCH wrote:
I am on MacOS, it is a development phase, but deployment will be on Linux.
It is hard to run dtrace/dtruss on MacOS due to the "system integrity protection" on MacOS. I got some output but very limited information.
So if there is a workaround you might suggest?
Why bother debugging it on macOS it never has to work there? If you are targetting linux then develop and test in linux I'd recommend. I use VMware fusion on the mac and run VMs for the target environements I need. Works very well. Barry
TS
On Thu, May 21, 2020 at 10:20 PM Jean-Paul Calderone <
exarkun@twistedmatrix.com> wrote:
On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH <
touch.sereysethy@gmail.com> wrote:
Hi Jean-Paul,
I found the error after working on a short, self contained, correcte example. It is not about twisted and normal thread in general, but something else.
After more tests, I come to realise that whenever I instantiate this class PolicyClient from this library https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in my protocol or just anywhere, twisted app either freezes or terminates, and this *only* happens when the app runs as a daemon, created using a service (.tac file).
I hope you can take a look at the class PolicyClient, why it causes problem to reactor main loop. Please point me to where the idea might come from.
I can give you the example but not sure if it is enough, as the client needs to connect to a server.
It looks like PolicyClient does a blocking HTTP request in __init__. Since PolicyClient is instantiated in the reactor thread, this will block the reactor. Perhaps this call is hanging for some reason when the process has daemonized? This would be unusual but not unheard of. For example, there are macOS environments where a process does not have access to the network if it is not associated with a windowing session.
What platform do you observe the problem on, and what more can you learn about what the process does on its way to hanging (eg can you run strace on it)?
Jean-Paul
Thank you, TS
On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
exarkun@twistedmatrix.com> wrote:
On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
touch.sereysethy@gmail.com> wrote:
Hello,
I am developing a twisted app, and it runs as a service using twistd -y to start the app. I am having a problem of running a thread. I know it is not recommended to use thread, but the library that I use, the object created is running in a thread.
Here is the problem:
1) if I start reactor by running reactor.run() directly, thread is running fine 2) if I run it as a service using twisted, the thread is not running, it runs but it seems to be blocked, because I tried to write something to file using time.sleep(), but file is empty.
Is there something that I miss? How can I debug this?
A good next step would be to create at Short, Self Contained, Correct (Compilable), Example <http://sscce.org/> and share it.
Jean-Paul
Thank you,
TS _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi, After I tested it on Linux, everything works fine. So I think the problem is related to MacOS restriction. I already reported the problem to the maintainers, because the library is supposed to work on both Linux and MacOS. But this new thread is killing the twisted main reactor. By the way I have another question regarding ssh client. I developed an ssh client, after authentication is made, I dont want to open a channel directly in serviceStarted in connection, I want to keep the connection so that I can use to open channel for doing exec later, my question is where should I keep that connection? Currently in my transport I add a defer in which it will be later called to create another object that I control, the defer is called back in serviceStarted, which I pass the connection object. I am not sure what is a correct way to do it. For now I feel like it is a cycling reference: transport -> connection -> transport -> my object to control connection. Thanks, TS On Fri, May 22, 2020 at 6:16 PM Barry Scott <barry.scott@forcepoint.com> wrote:
On Thursday, 21 May 2020 21:45:44 BST Sereysethy TOUCH wrote:
I am on MacOS, it is a development phase, but deployment will be on Linux.
It is hard to run dtrace/dtruss on MacOS due to the "system integrity protection" on MacOS. I got some output but very limited information.
So if there is a workaround you might suggest?
Why bother debugging it on macOS it never has to work there?
If you are targetting linux then develop and test in linux I'd recommend. I use VMware fusion on the mac and run VMs for the target environements I need. Works very well.
Barry
TS
On Thu, May 21, 2020 at 10:20 PM Jean-Paul Calderone <
exarkun@twistedmatrix.com> wrote:
On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH <
touch.sereysethy@gmail.com> wrote:
Hi Jean-Paul,
I found the error after working on a short, self contained, correcte example. It is not about twisted and normal thread in general, but something else.
After more tests, I come to realise that whenever I instantiate this class PolicyClient from this library
in my protocol or just anywhere, twisted app either freezes or terminates, and this *only* happens when the app runs as a daemon, created using a service (.tac file).
I hope you can take a look at the class PolicyClient, why it causes problem to reactor main loop. Please point me to where the idea might come from.
I can give you the example but not sure if it is enough, as the client needs to connect to a server.
It looks like PolicyClient does a blocking HTTP request in __init__. Since PolicyClient is instantiated in the reactor thread, this will block the reactor. Perhaps this call is hanging for some reason when the process has daemonized? This would be unusual but not unheard of. For example, there are macOS environments where a process does not have access to
https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py the
network if it is not associated with a windowing session.
What platform do you observe the problem on, and what more can you learn about what the process does on its way to hanging (eg can you run strace on it)?
Jean-Paul
Thank you, TS
On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
exarkun@twistedmatrix.com> wrote:
On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
touch.sereysethy@gmail.com> wrote:
Hello,
I am developing a twisted app, and it runs as a service using twistd -y to start the app. I am having a problem of running a thread. I know it is not recommended to use thread, but the library that I use, the object created is running in a thread.
Here is the problem:
1) if I start reactor by running reactor.run() directly, thread is running fine 2) if I run it as a service using twisted, the thread is not running, it runs but it seems to be blocked, because I tried to write something to file using time.sleep(), but file is empty.
Is there something that I miss? How can I debug this?
A good next step would be to create at Short, Self Contained, Correct (Compilable), Example <http://sscce.org/> and share it.
Jean-Paul
Thank you,
TS _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 5/19/2020 4:54 AM, Barry Scott wrote:
On Saturday, 16 May 2020 02:14:57 BST Ian Haywood wrote:
On 16/05/2020 10:55 am, Ian Haywood wrote:
y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega <wsanchez@wsanche <mailto:wsanchez@wsanchez.net>
I think it's great to get an SMB implementation in the Twisted org, but why would we even consider adding something like this to the main Twisted project? The advantage of twisted itself is cross-protocol abstractions such as cred. Of course you don't have to be in the repo to use them, but in
On 15/05/2020 10:52 am, Glyph wrote: practice developers need the discipline of a single project to maintain consistency, otherwise the human tendency to reinvent wheels is too strong Apropos we have two APIs for exporting filesystems which are broadly similar: SFTP in conch and FTP itself, SMB is about to be a third, ideally we should have one or have them descend from one another. It's slightly harder than it sounds as SMB is a file-access, as opposed to file-transfer, protocol and supports some extra features such as locking. I do not see a common class tree works for things that are so different as SMB and FTP.
Barry
I don't think the commonality lies on the network side of things. I think it is on the data manipulation side and the local file read/write side, which are currently inherantly synchronous unless you use threads and all that entails. All three protocols have to open existing files, create new files, read from files, write to files, read file metadata (dates, sizes, etc.), write metadata, traverse directory trees and packetize and send the various types of data and recieve and depacketize the data. Only the packetizing and depacketizing items are significantly different between SMB and the other two protocols. (SMB access to file data can be random or sequential, wherease FTP and SFTP are invariably sequential, but that should easily be accomodated.) -- John Santos Evans Griffiths & Hart, Inc. 781-861-0670 ext 539
On May 19, 2020, at 1:54 AM, Barry Scott <barry.scott@forcepoint.com> wrote:
I do not see a common class tree works for things that are so different as SMB and FTP.
Let's absolutely not try to do this with inheritance :-). Broadly, I agree with John Santos's assessment. But it's all sort of moot until someone starts trying to actually do it and we can talk about a specific proposal! -g
participants (7)
-
Barry Scott -
Chris Withers -
Glyph -
Ian Haywood -
Jean-Paul Calderone -
John Santos -
Sereysethy TOUCH