But none of these seem to allow asynchronous calls, which make a huge difference when dealing with a large structure. Nothing I've found really does what I need and I'm trying to keep my dependency list short so I'll end up writing something myself anyway, but my real question was whether this is something that could usefully be included in the stdlib. Having an object-oriented / polymorphic file API means that one can

provide file objects that aren't backed by the filesystem, but look the same. This is cool in that you can do things like treat zip files as directory, or mock out the filesystem for a unit test, without worrying about monkeypatching builtins or using a nonstandard wrapper API etc.

Great use case! And using factories to create create the objects would make this especially powerful.

That library, pyfilesystem has some serious firepower. On Sep 14, 2011 1:33 AM, "David Townshend" wrote:

But none of these seem to allow asynchronous calls, which make a huge

difference when dealing with a large structure. Nothing I've found really does what I need and I'm trying to keep my dependency list short so I'll end up writing something myself anyway, but my real question was whether this is something that could usefully be included in the stdlib.

...

Having an object-oriented / polymorphic file API means that one can provide file objects that aren't backed by the filesystem, but look the same. This is cool in that you can do things like treat zip files as directory, or mock out the filesystem for a unit test, without worrying about monkeypatching builtins or using a nonstandard wrapper API etc.

Great use case! And using factories to create create the objects would

make this especially powerful.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Wed, Sep 14, 2011 at 3:49 PM, David Townshend wrote:

Thanks for the pointer - PyFilesystem looks perfect for my requirements! I've had a look at PEP 355 and can see the drawbacks to the proposed implementation (especially subclassing str), but I'm not clear on what the problems are with the concept.

There's nothing wrong with the concept of a more object-oriented interface to the filesystem - PEP 355 was rejected due to the specifics of the proposed API rather than the idea in general being unacceptable. However, designing a nice OO filesystem API, getting feedback on it, getting it to a point where it is evolving slowly enough to be a suitable for inclusion in the stdlib, then getting it through the gauntlet of python-dev's design and implementation critique for standard library inclusion isn't exactly a task for the faint-hearted :)

 Would something like PyFilesystem be any more palatable?

I don't know the PyFilesystem API well enough to really say. However, from my quick review of the docs, it has potential. It's API oriented approach definitely aligns well with the role of the standard library in other areas (such as the file-like API itself, as well as more formal interfaces like the DB API and the crypto component APIs). I've cc'ed Ryan on a couple of messages in this thread, so hopefully he'll be inclined to chime in. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Thu, Sep 15, 2011 at 5:55 AM, Greg Ewing wrote:

On 14/09/11 03:03, David Townshend wrote:

3. Allow caching of file attribute data so that queries do not have to

...

wait the disk or network to respond (although at the cost of accuracy). ... The third can be met be allowing all disk calls to be asynchonous:

You're mixing up two completely different concepts here. Cacheing has nothing to do with asynchronous calls; it's storing the result so that you don't have to wait the *next* time you want the information.

I'm not sure if I'm mixing up concepts or terminology. My meaning is, for example, if a method Path.files() is used to obtain a list of files in a directory, it would call os.listdir() in another thread and store the result to a cache. At the same time, the current contents of the cache are returned by Path.files(). Most of the time, the cache would be written to after its contents are returned by Path.files(), so the actual value returned would be inaccurate, but would be more accurate on the next call. To me, this means caching the result of an asynchronous call to os.listdir(). I suspect, however, that I'm not using the term "asynchronous" as it normally refers to disk operations, so that's probably the confusion.

Both could be useful, but only for certain applications, and they should both be off by default in any general-purpose impelentation.

Agreed!