<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 31, 2015, at 5:27 PM, Brett Cannon <<a href="mailto:brett@python.org" class="">brett@python.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""><br class=""><div class="gmail_quote">On Sat Jan 31 2015 at 4:43:50 PM Donald Stufft <<a href="mailto:donald@stufft.io" class="">donald@stufft.io</a>> wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><blockquote type="cite" class=""><div class="">On Jan 31, 2015, at 4:22 PM, Brett Cannon <<a href="mailto:brett@python.org" target="_blank" class="">brett@python.org</a>> wrote:</div><br class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><br class=""><br class=""><div class="gmail_quote">On Sat Jan 31 2015 at 12:28:07 PM Donald Stufft <<a href="mailto:donald@stufft.io" target="_blank" class="">donald@stufft.io</a>> wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><blockquote type="cite" class=""><div class="">On Jan 31, 2015, at 12:00 PM, Brett Cannon <<a href="mailto:brett@python.org" target="_blank" class="">brett@python.org</a>> wrote:</div><br class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><br class=""><br class=""><div class="gmail_quote">On Sat Jan 31 2015 at 11:43:55 AM Donald Stufft <<a href="mailto:donald@stufft.io" target="_blank" class="">donald@stufft.io</a>> wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><blockquote type="cite" class=""><div class="">On Jan 31, 2015, at 11:31 AM, Brett Cannon <<a href="mailto:brett@python.org" target="_blank" class="">brett@python.org</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><br class=""><br class=""><div class="gmail_quote">On Sat Jan 31 2015 at 10:54:22 AM Paul Moore <<a href="mailto:p.f.moore@gmail.com" target="_blank" class="">p.f.moore@gmail.com</a>> wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">On 31 January 2015 at 15:47, Donald Stufft <<a href="mailto:donald@stufft.io" target="_blank" class="">donald@stufft.io</a>> wrote:<br class="">>> It's certainly possible to add a new API that loads resources based on<br class="">>> a relative name, but you'd have to specify relative to *what*.<br class="">>> get_data explicitly ducks out of making that decision.<br class="">><br class="">> data = __loader__.get_bytes(__name__, “logo.gif”)<br class=""><br class="">Quite possibly. It needs a bit of fleshing out to make sure it doesn't<br class="">prohibit sharing of loaders, etc, in the way Brett mentions.</blockquote><div class=""><br class=""></div><div class="">By specifying the package anchor point I don't think it does.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">Also, the<br class="">fact that it needs __name__ in there feels wrong - a bit like the old<br class="">version of super() needing to be told which class it was being called<br class="">from.</blockquote><div class=""><br class=""></div><div class="">You can't avoid that. This is the entire reason why loader reuse is a pain; you **have** to specify what to work off of, else its ambiguous and a specific feature of a specific loader.</div><div class=""><br class=""></div><div class="">But this is only an issue when you are trying to access a file relative to the package/module you're in. Otherwise you're going to be specifying a string constant like 'foo.bar'.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">But in principle I don't object to finding a suitable form of<br class="">this.<br class=""><br class="">And I like the name get_bytes - much more explicit in these Python 3<br class="">days of explicit str/bytes distinctions :-)</blockquote><div class=""><span style="font-size: 13.1999998092651px;" class=""><br class=""></span></div><div class=""><span style="font-size: 13.1999998092651px;" class="">One unfortunate side-effect from having a new method to return bytes from a data file is that it makes get_data() somewhat redundant. If we make it get_data_filename(package_name, path) then it can return an absolute path which can then be passed to get_data() to read the actual bytes. If we create importlib.resources as Donald has suggested then all of this can be hidden behind a function and users don't have to care about any of this, e.g. importlib.resources.read_data(module_anchor, path).</span><br class=""></div></div></div></div></blockquote><div class=""><br class=""></div></div></div><div style="word-wrap: break-word;" class=""><div class=""><div class="">I think we actually have to go the other way, because only some Loaders will be able to actually return a filename (returning a filename is basically an optimization to prevent needing to call get_data and write that out to a temporary directory) but pretty much any loader should theoretically be able to support get_data.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Why can only some loaders return a filename? As I have said, loaders can return an opaque string to simulate a path if necessary.</div></div></div></div></blockquote><div class=""><br class=""></div></div></div><div style="word-wrap: break-word;" class=""><div class=""><div class="">Because the idea behind get_data_filename() is that it returns a path that can be used regularly by APIs that expect to be handed a file on the file system.</div></div></div></blockquote><div class=""><br class=""></div><div class="">In my head that expectation is not placed on the method.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">Simulating a path with an opaque string isn’t good enough because, for example, OpenSSL doesn’t know how to open /data/foo.zip/foobar/cacert.pem. The idea here is that _if_ a regular file system path is available for a particular resource file then Loader().get_data_filename() would return it, otherwise it’d return None (or not exist at all).</div><div class=""><br class=""></div><div class="">This means that pkgutil.get_data_filename (or importlib.resources.get_filename) can attempt to call Loader().get_data_filename() and just return that path if one exists on the file system already, and if it doesn’t then it can create a temporary file and call Loader.get_data() and write the data to that temporary file and return the path to that.</div></div></div></blockquote><div class=""><br class=""></div><div class="">See I'm not even attempting to guarantee there is any API that will return a reasonable file system path as the import API makes no such guarantees. If an API like OpenSSL requires a file on the filesystem then you will have to write to a temporary file and that's just life. That's the same as if everything was stored in a zip file anyway.</div></div></div></div></blockquote><div class=""><br class=""></div></div></div><div style="word-wrap: break-word;" class=""><div class=""><div class=""><div class="">The entire *point* is this thread is that sometimes you need a file path that is a valid path to a resource.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Right, but I also have to make sure the import API doesn't get too ridiculous because it took me years and several versions of Python to make it work with the APIs inherited from PEP 302 and to make sure it grow into a huge mess.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><div class=""><br class=""></div><div class="">The naive approach is to just make it do something like:</div><div class=""><br class=""></div><div class=""><div class=""># in pkgutil</div><div class="">def get_data_filename(package, resource):</div><div class=""> <span class="Apple-converted-space"> </span>data = get_data(package, resource)</div><div class=""> <span class="Apple-converted-space"> </span>if data is not None:</div><div class=""> <span class="Apple-converted-space"> </span>with open("/tmp/path", "wb") as fp:</div><div class=""> <span class="Apple-converted-space"> </span>fp.write(data)</div><div class=""> <span class="Apple-converted-space"> </span>return "/tmp/path"</div><div class=""><br class=""></div><div class="">However the problem with this is that it imposes a read() into memory and then creating a new file, and then writing that data back to a file even in cases where there is already a file available on the file system. The Loader().get_data_filename() exists for a Loader() to *optionally* say that “We already have a file path for this file, so you can just use this instead of copying to a temporary location”.</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">And that's fine, but my point is forcing it to only play that role seems unnecessary. If you want a 'real' parameter to say "only return a path if I can pass it to an API that requires it" then that's fine.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><div class=""><div class=""><br class=""></div><div class="">Then the “optimized” but still naive approach becomes:</div><div class=""><br class=""></div><div class=""><div class=""># in pkgutil</div><div class="">def get_data_filename(package, resource):</div><div class=""> <span class="Apple-converted-space"> </span>mod = importlib.import_module(package)</div><div class=""> <span class="Apple-converted-space"> </span>if hasattr(mod.__loader__, "get_data_filename"):</div><div class=""> <span class="Apple-converted-space"> </span>try:</div><div class=""> <span class="Apple-converted-space"> </span>filename = mod.__loader__.get_data_filename(package, resource)</div><div class=""> <span class="Apple-converted-space"> </span>except FileNotFoundError:</div><div class=""> <span class="Apple-converted-space"> </span>pass</div><div class=""> <span class="Apple-converted-space"> </span>else:</div><div class=""> <span class="Apple-converted-space"> </span>if filename is not None:</div><div class=""> <span class="Apple-converted-space"> </span>return filename</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>data = get_data(package, resource)</div><div class=""> <span class="Apple-converted-space"> </span>if data is not None:</div><div class=""> <span class="Apple-converted-space"> </span>with open("/tmp/path", "wb") as fp:</div><div class=""> <span class="Apple-converted-space"> </span>fp.write(data)</div><div class=""> <span class="Apple-converted-space"> </span>return "/tmp/path"</div></div><div class=""><br class=""></div><div class="">This means there’s basically no penalty for using this API to access resources files when you’re accessing files from a FileLoader.</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">And leaking a temp file until shutdown which is why Barry and I prefer a context manager. =)</div></div></div></div></blockquote><div><br class=""></div><div>So the top level API can have both and people can use whichever fits their situation best.</div><div><br class=""></div><div>It’s not really leaking a temp file, it’s making it available to the process once the function has been called. This is a common use case to need a data file for the entire life of the process.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><div class=""><div class="">In my opinion anything that is harder to use than:</div><div class=""><br class=""></div><div class="">MY_PATH = os.path.join(os.path.dirname(__file__), “my/file.txt”)</div><div class=""><br class=""></div><div class="">Is highly unlikely to be used. People can already just write things to a temporary directory using get_data, but the point is they don’t because it’s a waste of time for the common case and it’s easier not to do that.</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">That's fine, but I also feel like we are trying to design around bad API design where something is assuming all data is going to be on disk and thus it's okay to require a file path on the filesystem instead of taking the bytes directly or a file-like object.</div><div class=""><br class=""></div><div class="">I realize you are trying to solve this specifically for OpenSSL since it has the nasty practice of wanting a file path, but from an import perspective I have to also worry about what makes sense for the API as a whole and from the perspective of import.</div></div></div></div></blockquote><div><br class=""></div><div>I’m not actually trying to solve this specifically for OpenSSL at all, I’m trying to solve it for any API that requires a file path where I don’t control that API. My end goal is to make it so zip imports are useful enough people can assume they are going to work, not a curiosity that mostly only works by accident for most projects. Right now you take almost any project on PyPI that has a data file and there’s an extremely high chance that it won’t work with zip import.</div><div><br class=""></div><div>My long term goals here are to make a “static” deployment format for Python that can wrap everything up into one file, so one step along this path is getting people to stop doing things that rely on having a real filesystem and only go through some abstraction. However I can’t get them to actually do that if the API to do that is awkward and something they don’t want to actually use. Purity is a great thing, but when there is a direct competitor to this API you have to weigh purity against actual usefulness in the common case for people.</div><div><br class=""></div><div>So yea, it’s designing around a bad API, but it’s also an API design that exists all over the place in the real world and telling people “well just don’t do that” means they won’t use this API and their thing will continue to not be usable with zip import.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class="">I think it is redundant but given that it’s a new API (passing module and a “resource path”) I think it makes sense. The old get_data API can be deprecated but left in for compatibility reasons if we want (sort of like Loader().load_module() -> Loader().exec_module()).</div></div></div></blockquote><div class=""><br class=""></div><div class="">If we do that then there would have to be a way to specify how to read the bytes for the module code itself since get_data() is used in the implementation of import by coupling it with get_filename() (which is why I'm trying not have to drop get_filename()/get_data() and instead come up with some new approach to reading bytes since the current approach is very composable). So get_bytes() would need a way to signal that you don't want some data file but the bytes for the module. Maybe if the path section is unspecified then that's a signal that the module's bytes is wanted and not some data file?</div></div></div></div></blockquote><div class=""><br class=""></div></div></div><div style="word-wrap: break-word;" class=""><div class=""><div class="">Perhaps trying to read modules and resource files with the same method is the wrong approach?</div></div></div></blockquote><div class=""><br class=""></div><div class="">If we are going to do that then we might as well deprecate all the methods that try to expose reading data and paths as the PEP 302 APIs tried to expose it uniformly.</div></div></div></div></blockquote><div class=""><br class=""></div></div></div><div style="word-wrap: break-word;" class=""><div class=""><div class="">I don’t think it makes sense to expose it uniformly, code is semantically different than data files and people need the ability to do different things with them. It’s unlikely you’ll get a 2GB.py file, however a 2GB data file is completely within the realms of possibility.</div></div></div><div style="word-wrap: break-word;" class=""><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class="">Maybe instead we should do: <a href="https://bpaste.net/show/b25b7e8dc8f0" target="_blank" class="">https://bpaste.net/show/b25b7e8dc8f0</a></div></div></div></blockquote><div class=""><br class=""></div><div class="">That seems like a bit much, e.g. why do you needs bytes **and** and a file-like object() when you get the former from the latter? And why do you need the path argument when you can get the path off the file-like object if it's an actual file object?</div></div></div></div></blockquote><div class=""><br class=""></div></div></div><div style="word-wrap: break-word;" class=""><div class=""><div class="">I don’t think it’s a bit much at all.</div><div class=""><br class=""></div><div class="">You get a stream method because sometimes things expect a file like object or sometimes the file is big and the ability to access a stream that handles that for you is super important. However when using a stream you need to ensure you close the stream after you’re done using it.</div></div></div></blockquote><div class=""><br class=""></div><div class="">With a context manager the closing requirement is negligible. And that only is an optimization if you're reading from something that allows for incremental reads, e.g. it's not an optimization for a SQL-backed loader (which is probably why PEP 302 has get_data() instead of get_file_object() or something).</div></div></div></div></blockquote><div><br class=""></div><div>In almost all uses where I would personally use it a context manager is awkward and I won’t use it and I’ll just continue to not be zip import compatible.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class="">You get a bytes method because sometimes you don’t care about all of that and you just need/want the raw bytes, it’s a nicer API for those people to be able to just get bytes without having to worry about reading a file or closing the file after they are done reading it.</div></div></div></blockquote><div class=""><br class=""></div><div class="">That seems unnecessary if you want to provide the optimization of allowing a file-like object to be returned when reading all of the bytes takes two lines of code instead of one. People know how to read files so it isn't like it's a new paradigm.</div></div></div></div></blockquote><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class="">You get a filename method because the stream method may or may not return a file object that has a path at all, and if you just need to pass the path into another API having an open file handle just to get the filename is a waste of a file handle.</div></div></div></blockquote><div class=""><br class=""></div><div class="">As I said above, I partially feel like the desire for this support is to work around some API decisions that are somewhat poor.</div><div class=""><br class=""></div><div class="">How about this: get_path(package, path, *, real=False) or get_path(package, filename, *, real=False) -- depending on whether Barry and me get our way about paths or you do, Donald -- where 'real' is a flag specifying whether the path has to work as a path argument to builtins.open() and thus fails accordingly (in instances where it won't work it can fail immediately and so loader implementers only have two lines of code to care about to manage it). Then loaders can keep their get_data() method without issue and the API for loaders only grew by 1 (or stays constant depending on whether we want/can have it subsume get_filename() long-term).</div><div class=""><br class=""></div><div class="">As for importlib.resources, that can provide a higher-level API for a file-like object along with some way to say whether the file must be addressable on the filesystem to know if tempfile.NamedTemporaryFile() may be backing the file-like object or if io.BytesIO could provide the API.</div><div class=""><br class=""></div><div class="">This gets me a clean API for loaders and importlib and gets you your real file paths as needed.</div><div class=""><br class=""></div></div></div></div></blockquote><div><br class=""></div><div>I honestly don’t really care what API the loader has because I don’t think anybody but the importlib.resources functions are ever going to use, so if you want to do something I hate with them knock yourself out I don’t care that much, I just think that requiring using __file__ and __path__ outside of the implementation of the loader itself is a pretty big code smell.</div><div><br class=""></div><div>How about this, instead here’s the top level APIs I want: <a href="https://bpaste.net/show/0c490aa07c07" class="">https://bpaste.net/show/0c490aa07c07</a></div><div><br class=""></div><div>How this is implemented in the Loader() API can be whatever folks want. The important thing is that these all solve actual use cases and solve them better and easier than the naive approach of using os.path functions directly. </div><div><br class=""></div><div>Important Things:</div><div><br class=""></div><div>* resource_filename and ResourceFilename() must return the real file system path if available and a temporary file else wise.</div><div>* resource_filename *must* be available for the lifetime of the process once the function has been called.</div><div>* ResourceFilename *must* clean itself up at the end of the context manager.</div><div>* These functions/context managers *must* work in terms of package names and relative file paths.</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><div class="">-Brett</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><div class="gmail_quote"><div class=""><br class=""></div><div class="">-Brett</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class="">This means that we’re not talking about “data” files, but “resource” files. This also removes the idea that you can call Loader.set_data() on those files (like i’ve seen in the implementation).</div></div></div><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""><span style="font-size: 13.1999998092651px;" class=""><br class=""></span></div><div class=""><span style="font-size: 13.1999998092651px;" class="">One thing to consider is do we want to allow anything other than filenames for the path part? Thanks to namespace packages every directory is essentially a package, so we could say that the package anchor has to encapsulate the directory and the path bit can only be a filename. That gets us even farther away from having the concept of file paths being manipulated in relation to import-related APIs.</span></div></div></div></div></blockquote><div class=""><br class=""></div></div></div><div style="word-wrap: break-word;" class=""><div class=""><div class="">I think we do want to allow directories, it’s not unusual to have something like:</div><div class=""><br class=""></div><div class=""><div class="">warehouse</div><div class="">├── __init__.py</div><div class="">├── templates</div><div class="">│ ├── accounts</div><div class="">│ │ └── profile.html</div><div class="">│ └── hello.html</div><div class="">├── utils</div><div class="">│ └── mapper.py</div><div class="">└── wsgi.py</div><div class=""><br class=""></div><div class="">Conceptually templates isn’t a package (even though with namespace packages it kinda is) and I’d want to load profile.html by doing something like:</div><div class=""><br class=""></div><div class="">importlib.resources.get_bytes(“warehouse”, “templates/accounts/profile.html”)</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Where I would be fine with get_bytes('warehouse.templates.accounts', 'profile.html') =)</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class="">In pkg_resources the second argument to that function is a “resource path” which is defined as a relative to the given module/package and it must use / to denote them. It explicitly says it’s not a file system path but a resource path. It may translate to a file system path (as is the case with the FileLoader) but it also may not (as is the case with a theoretical S3Loader or PostgreSQLLoader).</div></div></div></blockquote><div class=""><br class=""></div><div class="">Yep, which is why I'm making sure if we have paths we minimize them as they instantly make these alternative loader concepts a bigger pain to implement.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">How you turn a warehouse + a resource path into some data (or whatever other function we support) is an implementation detail of the Loader.</div></div></div><div style="word-wrap: break-word;" class=""><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""><span style="font-size: 13.1999998092651px;" class=""><br class=""></span></div><div class=""><span style="font-size: 13.1999998092651px;" class="">And just so I don't forget it, I keep wanting to pass an actual module in so the code can extract the name that way, but that prevents the __name__ trick as you would have to import yourself or grab the module from sys.modules.</span></div></div></div></div></blockquote><br class=""></div></div><div style="word-wrap: break-word;" class=""><div class=""></div><div class="">Is an actual module what gets passed into Loader().exec_module()?</div></div></blockquote><div class=""><br class=""></div><div class="">Yes.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class="">If so I think it’s fine to pass that into the new Loader() functions and a new top level API in importlib.resources can do the things needed to turn a string into a module object. So instead of doing __loader__.get_bytes(__name__, “logo.gif”) you’d do importlib.resources.get_bytes(__name__, “logo.gif”).</div></div></blockquote><div class=""><br class=""></div><div class="">If we go the route of importlib.resources then that seems like a reasonable idea, although we will need to think through the ramifications to exec_module() itself although I don't think there were be any issues.</div><div class=""><br class=""></div><div class="">And if we do go with importlib.resources I will probably want to make it available on PyPI with appropriate imp/pkgutil fallbacks to help people transitioning from Python 2 to 3.</div></div></div></div></blockquote></div></div><div style="word-wrap: break-word;" class=""><div class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div class="">---</div><div class="">Donald Stufft</div><div class="">PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA</div></div></div></div></div></blockquote></div></div></div></blockquote></div></div><div style="word-wrap: break-word;" class=""><br class=""><div class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div class="">---</div><div class="">Donald Stufft</div><div class="">PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA</div></div></div></div></div></blockquote></div></div></div></blockquote></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">---</div><div class="">Donald Stufft</div><div class="">PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA</div></div></div>
</div>
<br class=""></body></html>