Tell us what you did with Python this year....
As an attempt to generate some content and balance out the "jobs" discussion.... Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with.... I'm having a mad day but will try to post mine tonight or tomorrow... - Andy
2010/12/20 Andy Robinson <andy@reportlab.com>:
As an attempt to generate some content and balance out the "jobs" discussion....
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
Stub servers to simulate hardware behaviour with Twisted. Mock testing for a platform, that we're developing. -- Quiero ser el rayo de sol que cada día te despierta para hacerte respirar y vivir en me. "Favola -Moda".
On 20/12/10 12:43, Giorgio Zoppi wrote:
2010/12/20 Andy Robinson<andy@reportlab.com>:
As an attempt to generate some content and balance out the "jobs" discussion....
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with.... Stub servers to simulate hardware behaviour with Twisted. Mock testing for a platform, that we're developing.
Measuring clock error in a Software Defined Radio system (using gnuradio) in real time. Controlling a 2 axis mirror array to concentrate solar energy. Generating CAD files for the manufacture of a worm gear wheel. Making Christmas cards Sadly I only got paid for some of it. The invoices are generated in Python though. Dave Berkeley
On that topic, one of my highlights this year has been getting a ReportLab license. I'm now producing uninteresting letters with interesting tech; It's all about the journey not the destination I guess ;) On 20 Dec 2010, at 17:06, Andy Robinson wrote:
On 20 December 2010 14:43, <python@rotwang.co.uk> wrote:
Sadly I only got paid for some of it. The invoices are generated in Python though.
PDFs from ReportLab, I hope and trust? ;-)
-- Andy Robinson Managing Director ReportLab Europe Ltd. _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
On 20/12/10 17:06, Andy Robinson wrote:
On 20 December 2010 14:43,<python@rotwang.co.uk> wrote:
Sadly I only got paid for some of it. The invoices are generated in Python though. PDFs from ReportLab, I hope and trust? ;-)
No, but the Christmas cards are. The invoices are just dynamically generated HTML. I know they should be in PDF and I would certainly use your toolkit if they were. My backend is very primitive. D
Dear Mr Berkley, This sounds both fun and interesting! Tell me more! Was this business or pleasure? A small project of my own is to help my father with his milling machine. He wants to re-write a program in a more modern language: the program converts printed circuit board layouts into the milling machine's g-code language. At the moment the code is in an archaic Basic dialect. Is that sort of thing interesting to you? yours, Michael Grazebrook On 20/12/2010 14:43, python@rotwang.co.uk wrote:
Controlling a 2 axis mirror array to concentrate solar energy.
Generating CAD files for the manufacture of a worm gear wheel.
Hi Michael All I did was generate some DXF files that could be fed into a laser cutter. But I've never designed anything like that before, so the geometry and getting the gear teeth right was fun. It was for a business idea that didn't progress. Dave Berkeley On 20/12/10 18:02, Michael Grazebrook wrote:
Dear Mr Berkley,
This sounds both fun and interesting! Tell me more! Was this business or pleasure?
A small project of my own is to help my father with his milling machine. He wants to re-write a program in a more modern language: the program converts printed circuit board layouts into the milling machine's g-code language. At the moment the code is in an archaic Basic dialect.
Is that sort of thing interesting to you?
yours, Michael Grazebrook
On 20/12/2010 14:43, python@rotwang.co.uk wrote:
Controlling a 2 axis mirror array to concentrate solar energy.
Generating CAD files for the manufacture of a worm gear wheel.
_______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
OK, I'll bite, because what we're doing is exciting and new to me, so maybe to others also: Here at Resolver a couple of years ago we wrote a spreadsheet-like desktop application entirely in Python. Cell formulae are also full Python expressions, with a slightly augmented grammar to allow the use of common spreadsheet-isms, such as 'A1:C4' to refer to a cell range. This was specifically written in IronPython, with a Windows Forms GUI, because a number of our financial industry clients wanted .Net interoperability. We had a little trepidation about using IronPython, but in practice it turned out to work really well for us - the IronPython team did a really good job. Conversion between .Net and Python types is unobtrusively and automatically done, so that, for example, you don't have to worry about creating a .Net mapping type to pass to a static-typed C# API - you can simply pass it a regular Python dict, and everything magically 'just works' beautifully. In theory, we should have been able to run cross-platform using Mono, but to get started quickly, we used a third-party GUI component which made win32 calls. This made it impossible to run on other platforms, and with hindsight we should have fixed this already, by replacing that GUI component, but that's still to be done. This year we converted that desktop application into a web application called Dirigible. Think 'Google Docs' spreadsheet, but again with full Python expressions in cells, plus the ability for users to inject their own code into the recalculation process, for example to define functions, import third-party modules, iterate over cells, etc. This is running in CPython, as a Django app, on Amazon EC2 instances. We have already made it possible for clients with extremely large spreadsheets (which take hours to recalculate on Excel) to manually partition calculations to run in parallel across several EC2 instances. Soon we hope to make this process automatic, by examining the dependency graph (which cells depend on which other cells) and passing large stand-alone chunks of the graph to be recalculated on other machines. A lot of our core code, such as formula parsing and dependency calculation, didn't use any .Net at all, and so getting Dirigible running under CPython using code from Resolver One was straightforward. Nevertheless, personally I'd never done any Django nor Javascript, nor EC2 before, and the other guys on the team weren't massively experienced in these things either, so there was a concern about how quickly we could make progress. Happily, we managed to go from first conception ('mkdir Dirigible') to first beta release of Dirigible in two and a half months, and have been releasing updates every couple of days since, which is something I'm very proud of. Going forward, we hope that the two products will compliment each other (web apps aren't the right solution for everyone), and that code-sharing between the projects will mean that improvements in one (e.g. in providing robust statistical functions, or more Excel-compatible functions) will also improve the other. Interested to hear anyone else's stories. Jonathan Hartley Resolver Systems On 20/12/2010 12:18, Andy Robinson wrote:
As an attempt to generate some content and balance out the "jobs" discussion....
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
I'm having a mad day but will try to post mine tonight or tomorrow...
- Andy _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
-- Jonathan Hartley Made of meat. http://tartley.com tartley@tartley.com +44 7737 062 225 twitter/skype: tartley
On 20 Dec 2010, at 15:41, Jonathan Hartley wrote:
clients with extremely large spreadsheets (which take hours to recalculate on Excel)
*shudder* But that is what makes what you have done even more amazing :) I'm pretty sure spreadsheets of that level of complexity have a lot of subtle excel'isms that you must support and at least be able to do on your system.
to manually partition calculations to run in parallel across several EC2 instances. Soon we hope to make this process automatic, by examining the dependency graph (which cells depend on which other cells) and passing large stand-alone chunks of the graph to be recalculated on other machines.
A lot of our core code, such as formula parsing and dependency calculation, didn't use any .Net at all, and so getting Dirigible running under CPython using code from Resolver One was straightforward. Nevertheless, personally I'd never done any Django nor Javascript, nor EC2 before, and the other guys on the team weren't massively experienced in these things either, so there was a concern about how quickly we could make progress. Happily, we managed to go from first conception ('mkdir Dirigible') to first beta release of Dirigible in two and a half months, and have been releasing updates every couple of days since, which is something I'm very proud of.
Going forward, we hope that the two products will compliment each other (web apps aren't the right solution for everyone), and that code-sharing between the projects will mean that improvements in one (e.g. in providing robust statistical functions, or more Excel-compatible functions) will also improve the other.
That is all really really cool :) -Matt -- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Design | Zope/Plone Development and Consulting | Co-location | Hosting
On 20/12/2010 16:51, Matt Hamilton wrote:
On 20 Dec 2010, at 15:41, Jonathan Hartley wrote:
clients with extremely large spreadsheets (which take hours to recalculate on Excel) *shudder*
But that is what makes what you have done even more amazing :) I'm pretty sure spreadsheets of that level of complexity have a lot of subtle excel'isms that you must support and at least be able to do on your system.
Heh. Even as a spreadsheet-making company, we empathise with the shudder - they are brilliant ad-hoc investigation tools, but are often inappropriately used, simply because they are the one tool many people are most familiar with. My boss coined (or at least uses) the term 'frankensheet' :-) You are right to pinpoint Excel compatibility as our biggest demon - we have Excel import functionality for the desktop product, but not yet for the web app, and it's not 100% - we sometimes come across customer sheets which use functions we haven't implemented (we opted to implement them as-and-when we came across them, so as to avoid doing lots of work emulating corners of Excel which are rarely used.) Similarly, we do occasionally find Excel sheets using formula grammar that we can't yet cope with, but that is rare these days. The biggest outstanding concerns on this front are user-interface issues (an Excel sheet that has some formatting that we don't properly duplicate, making it look wonky or even unreadable) or sheets that use VBA - we haven't yet implemented anything to deal with that. Jonathan -- Jonathan Hartley Made of meat. http://tartley.com tartley@tartley.com +44 7737 062 225 twitter/skype: tartley
I've been parsing government data with BeautifulSoup and Scraperwiki, and working on visualising it with NetworkX and Matplotlib. -- The only thing worse than e-mail disclaimers...is people who send e-mail to lists complaining about them
As an attempt to generate some content and balance out the "jobs" discussion....
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
I'm having a mad day but will try to post mine tonight or tomorrow...
Great question, and a welcome one. I don't use Python as much as I thought I would when I started learning it. But it's been great fun to play around with the NLTK natural language processing library. I'm quite a language enthusiast, so using it to find word frequencies and such (which is just the tip of the iceberg) was fun. Unicode interoperability is a pain, though, and I find it depressing to work with in Python2.x, because it never seems to behave predictably. I still have no idea why tokenizing Hungarian text and tokenizing German text are not fundamentally the same operation, for instance, but I assume it has to do with the messiness of ASCII standards around the two languages. And since more than half of the languages I've studied require unicode compatibility, I've dropped Python entirely until NLTK goes Python3.
On 20/12/2010 16:08, Alec Battles wrote:
Unicode interoperability is a pain, though, and I find it depressing to work with in Python2.x, because it never seems to behave predictably. I still have no idea why tokenizing Hungarian text and tokenizing German text are not fundamentally the same operation
I have no idea why they're not: <code - untested> import codecs with codecs.open ("german.txt", "rb", encoding="utf8") as f: german_text = f.read () with codecs.open ("hungarian.txt", "rb", encoding="utf8") as f: hungarian_text = f.read () # do_stuff_with (german_text) # do_stuff_with (hungarian_text) </code> Of course, I'm assuming that you know what encoding has been used to serialise the text, but if you don't then it's not Python's fault ;) TJG
Unicode interoperability is a pain, though, and I find it depressing to work with in Python2.x, because it never seems to behave predictably. I still have no idea why tokenizing Hungarian text and tokenizing German text are not fundamentally the same operation
I have no idea why they're not:
<code - untested> import codecs
with codecs.open ("german.txt", "rb", encoding="utf8") as f: german_text = f.read ()
with codecs.open ("hungarian.txt", "rb", encoding="utf8") as f: hungarian_text = f.read ()
# do_stuff_with (german_text) # do_stuff_with (hungarian_text)
</code>
Of course, I'm assuming that you know what encoding has been used to serialise the text, but if you don't then it's not Python's fault ;)
Thanks. I'll try that. I seem to remember that 'file' in Linux detects encodings, but it's also a matter of calling it by the exact same name... Alec
On Mon, Dec 20, 2010 at 04:58:07PM +0000, Alec Battles wrote:
I seem to remember that 'file' in Linux detects encodings, but it's also a matter of calling it by the exact same name...
Any encoding detection will be heuristics; I've personally found the chardet library really useful when I'm stuck with a file that nobody knows the encoding of :) http://chardet.feedparser.org/ Cheers, Nick
Alec Battles wrote:
I seem to remember that 'file' in Linux detects encodings, but it's also a matter of calling it by the exact same name...
There is no foolproof way of detecting encoding unfortunately - you just need to know what it is before you read the file. Cheers, Doug. -- Telephone: +44 1904 567330, Mobile: +44 7879 423002 Switchboard: +44 1904 567349, Fax: +44 20 79006980 Post: Tower House, Fishergate, York, YO10 4UA, UK Registered in England. Company No 5171172. VAT GB843570325. Regd Office: 3&4 Park Court, Riccall Road, Escrick, York, YO19 6ED
I seem to remember that 'file' in Linux detects encodings, but it's also a matter of calling it by the exact same name...
There is no foolproof way of detecting encoding unfortunately - you just need to know what it is before you read the file.
That's interesting. I wonder if there's a mathematical proof of the 'undecidability' of text encodings.
On 20/12/10 17:53, Alec Battles wrote:
I seem to remember that 'file' in Linux detects encodings, but it's also a matter of calling it by the exact same name...
There is no foolproof way of detecting encoding unfortunately - you just need to know what it is before you read the file.
That's interesting. I wonder if there's a mathematical proof of the 'undecidability' of text encodings.
Hofstadter describes the problem in Godel, Escher, Bach as the "Envelope Problem" IIRC - you need to have some idea of how to decode any message you are sent, and you even need to understand that it is a "message". UNIX manages the latter for us by providing a filename - but how to interpret the contents is entirely up to you. It might be UTF-8, it might be a jpeg, it might be encrypted using AES. You need to know what to expect to try and interpret the contents. I bet there is a name for this (although probably not a proof), but I don't know what it is ;) Cheers, Doug. -- Telephone: +44 1904 567330, Mobile: +44 7879 423002 Switchboard: +44 1904 567349, Fax: +44 20 79006980 Post: Tower House, Fishergate, York, YO10 4UA, UK Registered in England. Company No 5171172. VAT GB843570325. Regd Office: 3&4 Park Court, Riccall Road, Escrick, York, YO19 6ED
2010/12/20 Doug Winter <doug.winter@isotoma.com>:
On 20/12/10 17:53, Alec Battles wrote:
I seem to remember that 'file' in Linux detects encodings, but it's also a matter of calling it by the exact same name...
There is no foolproof way of detecting encoding unfortunately - you just need to know what it is before you read the file.
That's interesting. I wonder if there's a mathematical proof of the 'undecidability' of text encodings.
Hofstadter describes the problem in Godel, Escher, Bach as the "Envelope Problem" IIRC - you need to have some idea of how to decode any message you are sent, and you even need to understand that it is a "message".
UNIX manages the latter for us by providing a filename - but how to interpret the contents is entirely up to you. It might be UTF-8, it might be a jpeg, it might be encrypted using AES. You need to know what to expect to try and interpret the contents.
I bet there is a name for this (although probably not a proof), but I don't know what it is ;)
You could give some heuristic on well known message domains as well, but it would lead to false negative or false positive. For example you could pattern match on file initial contents. This problem reminds me what we did several years ago on intrusion detection systems. Best Regards, Giorgio. -- Quiero ser el rayo de sol que cada día te despierta para hacerte respirar y vivir en me. "Favola -Moda".
On 20 Dec 2010, at 16:46, Tim Golden <mail@timgolden.me.uk> wrote:
On 20/12/2010 16:08, Alec Battles wrote:
I still have no idea why tokenizing Hungarian text and tokenizing German text are not fundamentally the same operation
Those languages have different grammatical structure inflexion and stemming rules amongst others. HTH
I have no idea why they're not:
<code - untested> import codecs
with codecs.open ("german.txt", "rb", encoding="utf8") as f: german_text = f.read ()
with codecs.open ("hungarian.txt", "rb", encoding="utf8") as f: hungarian_text = f.read ()
# do_stuff_with (german_text) # do_stuff_with (hungarian_text)
</code>
Of course, I'm assuming that you know what encoding has been used to serialise the text, but if you don't then it's not Python's fault ;)
TJG _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
I've used a lot of Python/Django for vehicle/resource tracking http://www.xlocate.net/ <http://www.xlocate.net/guided-tour/> - about 50% of the server-side is Python, the rest being Javascript/NodeJS. The web site uses Django CMS. I've also used Python a dozen times for one-off text processing jobs. - Chris On 20 December 2010 12:18, Andy Robinson <andy@reportlab.com> wrote:
As an attempt to generate some content and balance out the "jobs" discussion....
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
I'm having a mad day but will try to post mine tonight or tomorrow...
- Andy _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
On 20 Dec 2010, at 12:18, Andy Robinson wrote:
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
Well a couple highlights of my year: * I organised the Plone Conference 2010 which was held in the UK this year. We had nearly 300 attendees from 33 different countries. This was a fantastic event, a great community of people and demonstrating the power of python in larger systems * We launched a python based eMentoring site for Brightside UNIAID, with currently around 60 schemes (each a separate site), 18,000 registered users with 100 logged in per hour. * I was elected Vice President of the Plone Foundation. * We hired another python developer (bringing our small company up to 13 people in total) and gave everyone a 10% pay rise, bucking the doom and gloom of the recession. -Matt -- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Design | Zope/Plone Development and Consulting | Co-location | Hosting
On 20/12/10 17:00, Matt Hamilton wrote:
* I organised the Plone Conference 2010 which was held in the UK this year. We had nearly 300 attendees from 33 different countries. This was a fantastic event, a great community of people and demonstrating the power of python in larger systems
And very good it was too! Cheers, Doug. -- Telephone: +44 1904 567330, Mobile: +44 7879 423002 Switchboard: +44 1904 567349, Fax: +44 20 79006980 Post: Tower House, Fishergate, York, YO10 4UA, UK Registered in England. Company No 5171172. VAT GB843570325. Regd Office: 3&4 Park Court, Riccall Road, Escrick, York, YO19 6ED
Not sure if I'll allowed to discuss the day job, but in my off time I've been developing pyfilesystem (http://code.google.com/p/pyfilesystem/). I recently added the ability to open filesystems with a url, and created a bunch of command line applications which make use of that. This turned out to be an insanely useful and versatile feature (IMHO). Here are some examples of what you can do the command line apps: Display a file on an ftp server: fscat ftp://ftp.mozilla.org/README Serve an ftp site over http, effectively creating a ftp to http gateway: fsserve ftp://ftp.mozilla.org Display the directory structure of a zip file: fstree zip://myzip.zip Copy all the log files on an sftp server to a zip file on an ftp server: fscp sftp://will:password@example.org/logs/*.log zip: ftp://ftp.example.org/pub/archivedlogs.zip Mount a ram drive in your home directory fsmount ~/ramdrive mem:// Its all in the repo trunk now. I plan to release version 0.4 in January once I've written a shed load of docs... Will On Mon, Dec 20, 2010 at 12:18 PM, Andy Robinson <andy@reportlab.com> wrote:
As an attempt to generate some content and balance out the "jobs" discussion....
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
I'm having a mad day but will try to post mine tonight or tomorrow...
- Andy _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
-- Will McGugan http://www.willmcgugan.com
We've been building all sorts of stuff, much of which I can't talk about annoyingly. We're about 50% Plone, 50% other stuff - primarily Django, Twisted and plain old Python. Most of my time has been taken up with our largest client who is in the Pharmaceutical industry. We've done lots of cool stuff with buildout, which is a superb bit of software that should get more airtime outside Plone. We've written quite a few handy recipes: http://pypi.python.org/pypi?%3Aaction=search&term=isotoma We have some particularly mad stuff going with buildout and buildbot (buildout deploying buildbot that deploys buildout in VMs to deploy buildbot in one case!). I'd encourage anyone with production config management or release tasks using Python to strongly consider buildout. I have been working on pyspotify and squeal in my own time: https://github.com/winjer And I hope to find some more time for them over xmas. Also I've been playing with Arduino, although that's so straightforward to use Python does everything I need it to. I've fiddled with ZigBee networking, which is quite good fun and I've got some basic stuff for that in Python. Cheers, Doug. -- Telephone: +44 1904 567330, Mobile: +44 7879 423002 Switchboard: +44 1904 567349, Fax: +44 20 79006980 Post: Tower House, Fishergate, York, YO10 4UA, UK Registered in England. Company No 5171172. VAT GB843570325. Regd Office: 3&4 Park Court, Riccall Road, Escrick, York, YO19 6ED
Also I've been playing with Arduino, although that's so straightforward to use Python does everything I need it to. I've fiddled with ZigBee networking, which is quite good fun and I've got some basic stuff for that in Python.
Cheers,
Doug.
Hi Doug, I went to a talk by you (on TurboGears?) in 2007 just before I moved out of York. I hope things are going well. I was working on a mad project at the time to analyse rail journeys to determine which single staged fares could save money. The UK rail system has the most bizarre fares structure. You can exploit it by breaking the route into sections and buying tickets for each step. On some routes you can regularly save 40% or more on the fare. We pulled the plug as we realised that we would run into legal problems. I should have open sourced it at the time. We used Beautiful Soup to gather the data and used a distributed network using Twisted (we never used the word botnet) to disguise the source of the requests. The code worked fine but it was not a good project to pursue. I did some Zigbee work in 2006 for a company in Cambridge, Alert Me. They were using it for home security systems, but seem to have shifted more to green energy monitoring. They used Python a lot too, on embedded Linux. The Zigbee specs were all in flux back then. The Arduino is a lovely little board. It is what I was using for motor control for my solar concentrator. I used Python and PyEphem to track the sun, or rather reflect the sun onto a target. I couldn't find a partner or funding so I had to get a proper job. The electronics is all very simple, but I'm not a Mech Eng. The idea is simple and not new. An array of mirrors to focus sunlight onto a single spot. Static panels only get a cosine of the available sunlight, to they lose out in morning and evening. If you track the sun with a mirror you effectively halve the angle to get useful energy all day. Mirrors are much cheaper than solar panels. Most work has been done on big installations with expensive mirrors, massive motors, big concrete mounts. My approach was to use small cheap lightweight mirrors, more of them, each controlled by a micro, fixed to existing structures (eg. a wall). The prototype used the Arduino. As long as you know lon / lat and your current time you can calculate the Sun's position in the local sky. The difficult part is calibrating the mirrors. I was working on that when I ran out of money and had to get a "proper job". A single low powered processor board (something like the beagleboard) running linux talking to multiple cheap microcontrollers (eg. Arduino) would be able to control a sizeable array with very low power consumption. I hoped to target small installations in places with no fossil fuels, plenty of clear sky days and poor power infrastructure. Alternatively people with a property portfolio, eg. housing associations or developers. This guy is building really big installations and has very good funding : http://www.idealab.com/inside_the_lab/videos/esolar_tour.html I wanted to do the same thing on a small scale. It could make a big difference to a lot of communities. Dave Berkeley
Bit of this year and last... We have a 3rd-party COTS Helpdesk system with a woeful desktop interface and a naive [and that's being charitable] email ingest mechanism. All the data's stored in a SQL Server and is fairly clearly two different apps bolted together... A few years ago I knocked up a simple cherrypy web interface for it, mostly to avoid my having to use the egregious supplied UI. Then about 18 months ago I got so annoyed at the email ingest that I ran up an alternative as a proof of concept. At the same time I added an email alerting mechanism which sends out lightly-formatted emails on call updates. Now, several editions (and additions) later, the email ingest and alert mechanisms are easily the most used way to communicate with the IT Helpdesk. We encourage users to "communicate through the call" since it leaves the conversation trail visible to anyone who needs to look; it handles attachments such as screenshots of corrupts docs and is very easy to use. At the same time, the users see the assignments and Support activity so there's much greater transparency than before. A number of the ideas are certainly inspired by the concepts in Roundup (and possibly in other issue trackers) although I've never looked at the code there, and the concepts are similar, not identical. Key features: * Email ingest adequately determines the correct "conversation" to link to, handles embedded or linked attachments, and does a fairly good job of stripping out "noise". * Some actions can be handled entirely by email such as reassignment, status change, etc. although this isn't widely used. * Email alerting notifies on call creation, closure, update and a few specific status changes; includes attachment links and does some header munging so the originator appears to be the updater but the reply goes to the Helpdesk. * "Nosy" list concept akin to Roundup's where several people will receive call updates (even a distribution list...) * Master / child call -- this is quite recent and we're still playing with it, but basically a tree of calls can be closed together and the upper ones show the lower updates. The key requirement here is a split-assignment, such as when a new Blackberry is requested and the ordering is handled through the Facilities team while IT do the commissioning. Key tools / libraries: * Web interface: cherrypy, hand-crafted SQL, messy string-based HTML with %-formatting. [Not proud of this part; it's in real need of refactoring] * Email ingest: Exchange emapi, regular expressions for stripping out noise * Email alerts: smtplib, AD, jinja2 for formatted emails, regexes again for message manipulation Future ideas: * Alarms: eg allowing a call to "sleep" for an agreed period, to be picked up later; notifying when SLA milestones are passed etc. TJG
Hey ya, Enjoyed many London python dojos... and learnt a number of little tips from there, and met a bunch of london python people. That was probably the highlight of 2010 python involvement for me. Started writing a shit JavaScript interpreter (very early stages). ** Wrote a minimal new content management system(html as well as images, video, twitter, events and facebook related content). Did some billing software stuff. Started on a website library... pywebsite. ** Helped make a few mini games(including woger the wibbly wobbly wombat with a group from the london dojo). Wrote a few tools to help with the pygame website (which is still mostly php). ** Helped port a few things to py3k. Did a bunch of packaging improvements to a few libraries of mine, and some open source. ** Made some improvements to have python bindings for portmidi incorporated into the portmidi version control... as pyportmidi. ** Two video performances using pygame, numpy etc. A lot of that was using python to prepare/collect the data to display. ** worked on reducing memory used by python app servers(so I can run lots more little python apps on my cheap, low memory servers). Well, that's all I'm allowed to talk about now (it's not really interesting stuff under NDA anyway). cu. On Mon, Dec 20, 2010 at 12:18 PM, Andy Robinson <andy@reportlab.com> wrote:
As an attempt to generate some content and balance out the "jobs" discussion....
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
I'm having a mad day but will try to post mine tonight or tomorrow...
- Andy _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
For a while now I've been interested in sports / games 'ratings' systems. Some years back I collected results from weekly football games and wanted a way to score individual players from team score results (the teams change each week) - ultimately I wanted to help pick balanced teams). This year I set up a company table tennis ladder on racquetladder.com but was desperately unhappy with the ranking system - the winner / loser simply switch (or stick) ranks, which is terribly unstable (although very transparent). So I made a python / numpy implementation of Rémi Coulom's 'Whole History Rating' algorithm, and experimented a little with creating 'interactive' SVG plots of the results. Then I took part in the Planetwars "Google" AI challenge, and chose to use Python, thinking it would be dominated by clever heuristics rather than brute calculation. Top UK Python entry (*bow*) but, sadly, other languages dominated. Interestingly, they used Coulom's 'Bayeselo' to rate the entries, about which there was much griping, but nobody could think of a better way to do it -- I think generally people underestimated just how powerful Bayeselo is, and besides, those people were all over on kaggle.com where they concurrently had a 'design a better Elo' competition! So, for me it was an interestingly coincidental year. Oh, I also went to a London Dojo -- it was a lot of fun and I hope to go again. I don't really have any projects on the go at the moment so 2011 is a blank slate. Definitely want to win the next ai challenge :) Nice to have a chance to talk about these things. Matt
I haven't done much python this year, but had a great time at Europython and the London dojo. As I speak, a couple of days using jython for websphere automation awaits... On 21 December 2010 12:22, Matthew Turnbull <mattcturnbull@googlemail.com>wrote:
For a while now I've been interested in sports / games 'ratings' systems. Some years back I collected results from weekly football games and wanted a way to score individual players from team score results (the teams change each week) - ultimately I wanted to help pick balanced teams). This year I set up a company table tennis ladder on racquetladder.com but was desperately unhappy with the ranking system - the winner / loser simply switch (or stick) ranks, which is terribly unstable (although very transparent). So I made a python / numpy implementation of Rémi Coulom's 'Whole History Rating' algorithm, and experimented a little with creating 'interactive' SVG plots of the results.
Then I took part in the Planetwars "Google" AI challenge, and chose to use Python, thinking it would be dominated by clever heuristics rather than brute calculation. Top UK Python entry (*bow*) but, sadly, other languages dominated. Interestingly, they used Coulom's 'Bayeselo' to rate the entries, about which there was much griping, but nobody could think of a better way to do it -- I think generally people underestimated just how powerful Bayeselo is, and besides, those people were all over on kaggle.com where they concurrently had a 'design a better Elo' competition! So, for me it was an interestingly coincidental year.
Oh, I also went to a London Dojo -- it was a lot of fun and I hope to go again. I don't really have any projects on the go at the moment so 2011 is a blank slate. Definitely want to win the next ai challenge :)
Nice to have a chance to talk about these things. Matt _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
Hi Guys, Obviously I went to lots of London Python dojos... It's nice to see that the format is being used for other languages too (I know of at least Clojure and Scala dojos and plans for an Erlang one). We managed to finish our adventure game and plans are afoot for a new Pygame based project (see https://github.com/ntoll/PyMerryChristmas for a Christmas card from the dojo written with Pygame). The dojo has given me lots of learning opportunities and a chance for community building - so good to put names to faces/twitter accounts/IRC usernames etc... I've been using Python for just over two years now and I can safely say I'm having the most fun I've ever had as a developer. This year's highlights have been: * Delivering a large web-application built with Django for Fry-IT * Creating / collaborating on lots of FluidDB related clients / libraries: fluiddb.py, django-fluiddb, FOM (with Ali Afshar) and flimp. * Joining Fluidinfo full time has meant my first contact with Twisted. * Giving a hands-on Python workshop/tutorial to the NortHACKton guys. * Exploring the wider Python ecosystem. * Europython (at which I gave two talks and was great fun). Here's hoping 2011 is a good one for us all. Nicholas.
Hello all, I started the year working for a German firm writing web applications with Django on the server and Silverlight on the front end. The front end application (about 20 000 lines) was written entirely in IronPython and running in the browser. Communication with django was almost exclusively sending and receiving json. In September I started working for Canonical, working on web services associated with Ubuntu Pay, the new software centre and the single sign on services for canonical websites. This is mostly working with django. unittest in the Python standard library has continued to evolve, which I've been heavily involved with along with a few other standard library tweaks (my favourites being contextlib.ContextDecorator and inspect.getattr_static plus backporting weakref.WeakSet to Python 2.7). Python 2.7 was released in 2010 as well as the first alphas and betas of 3.2. The changes to unittest are available in a backport to earlier versions of Python called unittest2 which has gained a surprisingly large number of users and will be incorporated in the next version of django. My 'other' testing project is mock, which is approaching a 0.7.0 release with a lot of new features. The blocker on getting the final version out of the door is a thorough review of the documentation. Hopefully I will find time for this over the christmas break. The beta seems to be getting a nice level of uptake though and no new bug reports since the last release... ConfigObj is still reasonably popular but I'm not using it a great deal myself these days, so I haven't worked much on it this year. A couple of folk ported it to Python 3 though, which is fantastic. In 2010 I attended PyCon US, EuroPython and PyCon Italia. All were fantastic and at EuroPython this year we had our first language summit and PSF members meeting. I'm currently helping with talk selection for PyCon US 2011 and organising the language and vm summits we will have there. We intend to repeat the language summits and PSF members meeting at EuroPython 2011. The EuroPython 2011 location is fantastic, so hopefully see you all there! All the best, Michael Foord -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On 21/12/2010 14:45, Michael Foord wrote:
my favourites being contextlib.ContextDecorator
I didn't know that had your fingerprints on it! Nice one - I love this and use it all the time. Jonathan -- Jonathan Hartley Made of meat. http://tartley.com tartley@tartley.com +44 7737 062 225 twitter/skype: tartley
On 21 December 2010 14:56, Jonathan Hartley <tartley@tartley.com> wrote:
On 21/12/2010 14:45, Michael Foord wrote:
my favourites being contextlib.ContextDecorator
I didn't know that had your fingerprints on it! Nice one - I love this and use it all the time.
It came out of the pattern used in mock where patch (and other decorators) also work as context managers. This was first implemented at Resolver Systems... (I think it was Tom's idea.) All the best, Michael
Jonathan
-- Jonathan Hartley Made of meat. http://tartley.com tartley@tartley.com +44 7737 062 225 twitter/skype: tartley
_______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On 21 December 2010 18:09, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 21 December 2010 14:56, Jonathan Hartley <tartley@tartley.com> wrote:
On 21/12/2010 14:45, Michael Foord wrote:
my favourites being contextlib.ContextDecorator
I didn't know that had your fingerprints on it! Nice one - I love this and use it all the time.
It came out of the pattern used in mock where patch (and other decorators) also work as context managers. This was first implemented at Resolver Systems... (I think it was Tom's idea.)
Heh neat that has made it into the stdlib, even neater that it's used in contextlib.contextmanager by default. I had no idea! Though rather disappointing that it is no longer a cute hack to impress people with ;-) FWIW I'd first heard of it from Dave Beazley's blog, I guess it got invented in several places at the same time which must be a good sign too. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On 21/12/2010 18:28, Floris Bruynooghe wrote:
On 21 December 2010 18:09, Michael Foord<fuzzyman@voidspace.org.uk> wrote:
On 21 December 2010 14:56, Jonathan Hartley<tartley@tartley.com> wrote:
On 21/12/2010 14:45, Michael Foord wrote:
my favourites being contextlib.ContextDecorator I didn't know that had your fingerprints on it! Nice one - I love this and use it all the time. It came out of the pattern used in mock where patch (and other decorators) also work as context managers. This was first implemented at Resolver Systems... (I think it was Tom's idea.)
Heh neat that has made it into the stdlib, even neater that it's used in contextlib.contextmanager by default. I had no idea! Though rather disappointing that it is no longer a cute hack to impress people with ;-)
FWIW I'd first heard of it from Dave Beazley's blog, I guess it got invented in several places at the same time which must be a good sign too.
I think there was similar code in django and also py.test (making APIs that works as both context managers and decorators) - the fact that it was an established pattern than several people had found useful already is why we added it to the standard library. :-) Michael
Regards Floris
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Mon, Dec 20, 2010 at 12:18 PM, Andy Robinson <andy@reportlab.com> wrote:
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
I have worked this year on a web based data quality system written entirely in python. It is targeted towards business users. The system uses SQLAlchemy to reflect a database metadata and provides the users with a visual view of the database schema using django/jqGrid. It provides the users with ability to add rules and schedule them to run on specified periods. The rules can vary from regular expressions fields, sets of values, range of numbers, lookup values to another table or plain sql statements. The system can email alerts when a rule or a batch of rules identify bad data. It can also display charts that express data distribution using reportlab. Most of the web interface is done in an ajaxy way and django piston was big part of that. Happy holidays ... Meitham
On 20/12/2010 12:18, Andy Robinson wrote:
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with.... It's been an interesting year in Python for me. The London dojos have been fantastic - met some great people and learnt a lot of new skills. I also submitted some test coverage code to Python 3.2 while at EuroPython, which was cool.
Until recently, I was maintaining Python-based data feed systems and using a liberal application of Python to fix things that weren't easy to fix using other tech. I'm now involved with a new company and doing prototyping work for redeveloping a huge system written in PHP and Perl, replacing particular core components with Python, Django and Celery. Hurray! Two personal projects have occupied me this year. The first is a wargame written using PyGame - the code is a mess and has been neglected the last few months. I plan to rewrite most of it based on lessons learnt and release the source code properly (it's in a private BitBucket repo at the moment). The second project is FluidInYourEar, which uses FluidDB as the backend database. It's a music band/genre browser which ultimately wants to become a communal music recommender (mostly heavy metal but could be anything). I'm midway porting it to Flask and Google's AppEngine, but don't have time to progress it at the moment. I gave a talk in October on FIYE which was the first time I've given a tech talk - hope to do more such talks next year, including one at PyCon AU 2011 if I can pluck up enough courage. John
a wargame
Hey. What sort of game? 2d? Turn-based? Hex tiles? Or what? And are you using Pygame or pyglet or something else? On 21/12/2010 15:25, John Chandler wrote:
On 20/12/2010 12:18, Andy Robinson wrote:
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with.... It's been an interesting year in Python for me. The London dojos have been fantastic - met some great people and learnt a lot of new skills. I also submitted some test coverage code to Python 3.2 while at EuroPython, which was cool.
Until recently, I was maintaining Python-based data feed systems and using a liberal application of Python to fix things that weren't easy to fix using other tech. I'm now involved with a new company and doing prototyping work for redeveloping a huge system written in PHP and Perl, replacing particular core components with Python, Django and Celery. Hurray!
Two personal projects have occupied me this year. The first is a wargame written using PyGame - the code is a mess and has been neglected the last few months. I plan to rewrite most of it based on lessons learnt and release the source code properly (it's in a private BitBucket repo at the moment).
The second project is FluidInYourEar, which uses FluidDB as the backend database. It's a music band/genre browser which ultimately wants to become a communal music recommender (mostly heavy metal but could be anything). I'm midway porting it to Flask and Google's AppEngine, but don't have time to progress it at the moment. I gave a talk in October on FIYE which was the first time I've given a tech talk - hope to do more such talks next year, including one at PyCon AU 2011 if I can pluck up enough courage.
John
_______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
-- Jonathan Hartley Made of meat. http://tartley.com tartley@tartley.com +44 7737 062 225 twitter/skype: tartley
On 21/12/2010 15:33, Jonathan Hartley wrote:
a wargame
Hey. What sort of game? 2d? Turn-based? Hex tiles? Or what? And are you using Pygame or pyglet or something else? It's using PyGame. 2D, solitaire, turn-based, irregular area maps (sort of like Hearts of Iron II), with a bit of politics and management thrown in as well as combat.
I was learning to use PyGame and ran before I could walk. Now I know better, hence the need to revise the code :-) John
As an attempt to generate some content and balance out the "jobs" discussion....
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
I have automated an ill-thought process that used to involve creating an ad-hoc 1500 line windows batch program to carry out an unattended remote installation every time my company tried to deploy new content on their large network of Video Lottery Terminals. I was hired basically to help develop those silly windows batch programs which were creating a bottleneck, and by writing step by step a 5000 line python program that automatically generated the batch programs, the bottleneck went elsewhere. The whole process is as ill-thought as ever, but I'm no longer stressed, and they haven't had to replace the members of the team who left the company, effectively downsizing it from five to two people. I have also used Python to do lots of file processing, one time programs. Cheers Javier
On 21 December 2010 17:10, Javier Llopis <javier@correo.com> wrote:
As an attempt to generate some content and balance out the "jobs" discussion....
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
I have automated an ill-thought process that used to involve creating an ad-hoc 1500 line windows batch program to carry out an unattended remote installation every time my company tried to deploy new content on their large network of Video Lottery Terminals.
I was hired basically to help develop those silly windows batch programs which were creating a bottleneck, and by writing step by step a 5000 line python program that automatically generated the batch programs, the bottleneck went elsewhere. The whole process is as ill-thought as ever, but I'm no longer stressed, and they haven't had to replace the members of the team who left the company, effectively downsizing it from five to two people.
Yay for Python replacing people. Uhm, I think... ;-) Michael
I have also used Python to do lots of file processing, one time programs.
Cheers
Javier
_______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Mon, Dec 20, 2010 at 12:18 PM, Andy Robinson <andy@reportlab.com> wrote:
As an attempt to generate some content and balance out the "jobs" discussion....
Why don't a few people here tell us what they got up to this year? Neat projects at work, things you learned about Python in 2010, things you've been playing with....
I'm having a mad day but will try to post mine tonight or tomorrow...
At TweetDeck we've built our REST API on Twisted over Amazon EC2 and SimpleDB. We also rolled out a scheduled update (delay tweeting/updating in time to hit other timezones, etc), over RabbitMQ and Celery. We also continued extending our website, which is implemented in Django. Perhaps most interesting out of this is working both with synchronous and asynchronous code. We have common libraries implementing clients for dozens of REST apis that we use both on the Django and Twisted side. You usually have to choose one path or the other, but this year I've repurposed this common library to do either, depending on an instantiation argument. It turned out to not be so bad - but then why would it be? Most libraries out there blindly implement a synchronous model without realizing that splitting the request and response code is actually quite easy - and doing this makes it also easy to convert this to asynchronous code. # Usual synchronous code def get_foo(arg1, arg2): args = prepare_request() result = make_blocking_call() # usually uses something from httplib or httplib2 do_stuff(result) do_more_stuff(result) # async friendly version of the above def get_foo_nicer(arg1, arg2): args = prepare_request() result = make_maybe_blocking_call() def post_process(result): do_stuff(result) do_more_stuff(result) post_process(result) The second version is exactly the same as the previous, but now it becomes much easier to make it async (rather, Twisted aware). The key player to *making* this async-aware is the Deferred - that is, a "promise" that a result of the request will eventually appear. Then, you can make the last line slightly more intelligent by calling another method, which decides to magically apply the post_process function in the usual synchronous case, or add it as a callback to the deferred in the asynchronous case. I've implemented this pattern into boto in a "async aware" branch and it seems to work quite nicely, without disrupting much code. It strikes me a something that could easily be emulated in most other libraries Anyway, I thought that was mildly interesting and worth sharing. Merry Christmas everyone, and Happy New Year! Reza -- Reza Lotun mobile: +44 (0)7521 310 763 email: rlotun@gmail.com work: reza@tweetdeck.com twitter: @rlotun
Reza Lotun wrote:
# async friendly version of the above def get_foo_nicer(arg1, arg2): args = prepare_request() result = make_maybe_blocking_call() def post_process(result): do_stuff(result) do_more_stuff(result) post_process(result)
The second version is exactly the same as the previous, but now it becomes much easier to make it async (rather, Twisted aware). The key
Nice idea making your synchronous code work like this too. Have you considered using the @inlineCallbacks decorator instead? I bet you could make that work too, you could write your own decorator that uses inlineCallbacks or something synchronous, then the above becomes: @cleverInlineCallbacks def get_foo(arg1, arg2): args = prepare_request() result = yield make_maybe_blocking_call() do_stuff(result) finalValue = do_more_stuff(result) defer.returnValue(finalValue) Which can be a lot neater when you get stacks and stacks of callbacks. Python's lack of a block construct otherwise makes that really unpleasant. Cheers, Doug. -- Telephone: +44 1904 567330, Mobile: +44 7879 423002 Switchboard: +44 1904 567349, Fax: +44 20 79006980 Post: Tower House, Fishergate, York, YO10 4UA, UK Registered in England. Company No 5171172. VAT GB843570325. Regd Office: 3&4 Park Court, Riccall Road, Escrick, York, YO19 6ED
On Wed, Dec 22, 2010 at 11:38 AM, Doug Winter <doug.winter@isotoma.com> wrote:
Have you considered using the @inlineCallbacks decorator instead? I bet you could make that work too, you could write your own decorator that uses inlineCallbacks or something synchronous, then the above becomes: ...
Good point - I like inlineCallbacks for certain things like tests and initialization steps, but I try to avoid them in general because it's too easy to write problematic code with them. I actually fleshed out these reasons in this stackoverflow answer: http://stackoverflow.com/questions/3894278/twisted-deferred-addcallback-vs-y... Cheers, Reza -- Reza Lotun mobile: +44 (0)7521 310 763 email: rlotun@gmail.com work: reza@tweetdeck.com twitter: @rlotun
On 22/12/10 11:38, Doug Winter wrote:
[…] result = yield make_maybe_blocking_call() […]
This looks similar to Dave Beazley's use of coroutines to implement concurrency without threading. In Dave's case, the yielded expression is a request for the scheduler to perform some action at its leisure. When the scheduler has a result it's passed back in to the coroutine using the send() method and the coroutine continues on its way until the next yield. It's described in the PDF linked from <http://www.dabeaz.com/coroutines/>. The whole thing is worth a read (I think 'mind-bogglingly impressive' sums it up quite nicely), but if you want an example then the first instance of this I can find is on slide 135 (page 68). All the best, Alex
participants (27)
-
Alec Battles -
Alexander Dutton -
Alexander Harrowell -
Andy Robinson -
Chris Dew -
Doug Winter -
Edward Hartley -
Floris Bruynooghe -
Giorgio Zoppi -
Javier Llopis -
John Chandler -
Jonathan Hartley -
Kev Dwyer -
Matt Hamilton -
Matthew Turnbull -
meitham -
Michael Foord -
Michael Foord -
Michael Grazebrook -
Nicholas Tollervey -
Nick Murdoch -
python＠rotwang.co.uk -
René Dudfield -
Reza Lotun -
Rob Cowie -
Tim Golden -
Will McGugan