<blockquote style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;" class="gmail_quote">Indeed, I wish I had known about this a year ago; it would have saved
me a lot of work. Of course it probably didn't exist a year ago... :(
</blockquote><div><br>well, yeah. many people need "parsing-abilities", but they resort to ad-hoc parsers using struct/some ad-hoc implementation of their own. there clearly is a need for a generic, strong, and extensible parsing/building mechanism.
<br><br><blockquote style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;" class="gmail_quote">Well, declarative is less flexible. OTOH declarative is nice in the way it is more readable and allows more optimisations.
<br></blockquote></div><br>i don't think "less flexible" is the term. it's certainly different, but if you need something specific, you can always subclass a construct on your own. other than that, being declarative means easy to read/write/maintain/debug/upgrage (to a newer version of the library).
<br><br><blockquote style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;" class="gmail_quote"> IMHO, at least in theory Construct could have small but fast C
extension to take care of the encoding and decoding, which is the
critical path. Everything else, like the declaration part, can be
python, as it is usually done once on application startup.
<br></blockquote>well, i expected the encodings package to have a str.encode("bin") and str.decode("bin")... for some reason there's no such codec. it's a pity.<br><br><blockquote style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;" class="gmail_quote">
This is a very nice library indeed. But the number one feature that I
need in something like this would be to use C. That's because of my
application specific requirements, where i have observed that
reapeatedly using struct.pack/unpack and reading bytes from a stream
represents a considerable CPU overhead, whereas the same thing in C
would be ultra fast.<br></blockquote>well, you must have the notion of a "stream", i.e., go back and forth, be able to read/write bits/bytes at arbitrary locations, etc. i thought of moving the library to pyrex, and compiling it, but the number of critical parts is very small -- basically only the Repeater class could be improved by writing it in C.
<br>i mean, most of the time is consumed at creating objects in the objects tree, etc. for example, the Struct class simply iterates over the nested construsts and parses each of the in that sequence. doing a pythonic iteration of a C-level iteration over a pythonic object is practically the same.
<br><br><blockquote style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;" class="gmail_quote"> If you agree to go down this path I might even be able to volunteer some of my time to help, but it's not my decision.
<br></blockquote>well, mainly i'm looking for ideas. just moving it to c wouldnt be too helpful. <br>some ideas i have:<br>* making the stream work with bytes instead of bits, so that memory consumption would decrease 8-fold... but then parsing unaligned fields (either by size of position) is gonna be a headache
<br>* unifying the context tree with the parsing/building tree, to create less objects on the fly (but it has some issues)<br>* using lambda functions for meta expressions, instead of eval(string) -- perhaps it's faster, but lambda is getting deprecated by python3k :(
<br><br>apart from that, i'm rely on inheritance in many places. if some classes are written in C and some in python, i'm not sure how it could work (can a C class inherit a pythonic one? would it be easy to extend?). and, that means users would have to compile the C sources, while now all they have to do is extract a zip file. and then i'd have to write makefiles, and maintain those also... it's getting dirty. i like the painless "unzip-and-use" installation.
<br><br>so if you have ideas, i'd be happy to hear those.<br>thanks,<br><br>-tomer<br><br><div><span class="gmail_quote">On 4/18/06, <b class="gmail_sendername">Gustavo Carneiro</b> <<a href="mailto:gjcarneiro@gmail.com">
gjcarneiro@gmail.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="direction: ltr;"><div></div><div style="direction: ltr;">
<span class="q"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="direction: ltr;">why include Construct?<br>* the struct module is very nice, but very limited and non-pythonic as well
<br>* pure python (no platform/security issues)</div></blockquote></span></div><div style="direction: ltr;"><div><br> IMHO this is a drawback. More on this below.<br></div></div><div style="direction: ltr;"><span class="q">
<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="direction: ltr;">* lots of people need to parse and build binary data structures, it's not an esoteric library
<br>* license: public domain<br>* quite a large user base for such a short time (proves the need of the community)</div></blockquote></span></div><div style="direction: ltr;"><div><br> Indeed, I wish I had known about this a year ago; it would have saved me a lot of work. Of course it probably didn't exist a year ago... :(
<br> </div></div><div style="direction: ltr;"><span class="q"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="direction: ltr;">
* easy to use and extend (follows the componentization pattern)
<br>* declarative: you don't need to write executable code for most cases
</div></blockquote></span></div><div style="direction: ltr;"><div><br> Well, declarative is less flexible. OTOH declarative is nice in the way it is more readable and allows more optimisations.<br></div></div><div style="direction: ltr;">
<span class="q"><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="direction: ltr;">why not:<br>* the code is (very) young. stable and all, but less than a month on the loose. <br>* new features may still be added / existing ones may be changed in a non-backwards-compatible manner
<br><br>so why am i saying this now, instead of waiting a few months for it to maturet?
<br>well, i wanted to get feedback.
those of you who have seen/used the library, please tell me what you think:<br>* is it suitable for a standard library?</div></blockquote><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="direction: ltr;">* what more features would you want?<br>* any changes you think are necessary?</div></blockquote></span></div><div style="direction: ltr;"><div><br> This is a very nice library indeed. But the number one feature that I need in something like this would be to use C. That's because of my application specific requirements, where i have observed that reapeatedly using
struct.pack/unpack and reading bytes from a stream represents a considerable CPU overhead, whereas the same thing in C would be ultra fast.<br><br> IMHO, at least in theory Construct could have small but fast C extension to take care of the encoding and decoding, which is the critical path. Everything else, like the declaration part, can be python, as it is usually done once on application startup.
<br><br> If you agree to go down this path I might even be able to volunteer some of my time to help, but it's not my decision.<br> <br> Best regards.<br></div></div>
</div></blockquote></div><br>