Mailman 3 November 2011 - python-uk

London Python Code Dojo *this Thursday*
by Tim Golden Dec. 1, 2011

Dec. 1, 2011

Hi Folks, Apologies for the short notice. We were caught out by the fact that the first Thursday of the month is the first day of the month. (All right; *I* was caught out). The next London Python Code Dojo is happening this Thursday on the 1st December from 6.30pm. You can sign up here (tickets go quickly): https://ldnpydojo.eventwax.com/london-python-code-dojo-season-3-episode-4 It'll be at the offices of Fry-IT (who, at very short notice have stepped up to give us space and sponsor our pizza and beer). It'll be the usual (slightly newer) setup: beer & pizza at 6.30pm with the chance to put forward coding proposals for later. We'll start at 7.15pm with any lightning talks people may have -- please email me if you've got one you'd like to give. Then we vote on the week's coding challenge and spend a couple of hours group-coding it. At the end of the evening we show-and-tell, laugh at and with each other, and those who have any energy left head off to the pub. Where..? Fry-IT Ltd. Nearest Tubes: Waterloo Southwark Address: Fry-IT Limited 503 Enterprise House 1/2 Hatfields London SE1 9PG See you there! TJG

2 2

Next Python Sheffield Nov 29th
by Daley Chetwynd Nov. 27, 2011

Nov. 27, 2011

Hi all, The next Python Sheffield meeting is being held on Tuesday November 29th at the GIST Lab in Sheffield. The meeting will run from 19:00 - 20:45, although doors will be open at the GIST Lab from 18:30. The GIST Lab is located opposite Sheffield train station and behind the Showroom cinema: http://thegisthub.net/groups/gistlab<http://www.google.com/url?sa=D&q=http://thegisthub.net/groups/gistlab&usg=A…> This month we have the following speaker: Peter Russell (West Yorks Python) - "Things that make Python brilliant: First-class functions and decorators" We've currently got 11 people registered for the meeting. If you'd like to join us then please register for free at: http://pysheff1111.eventbrite.co.uk<http://www.google.com/url?sa=D&q=http://pysheff1111.eventbrite.co.uk&usg=AF…> We're always looking for future speakers, so if you'd like to talk on anything Python-related and fancy a trip over to Sheffield, please get in touch. To find out more about the Python Sheffield group, follow @pysheff on Twitter or see the Google group: http://groups.google.com/group/python-sheffield<http://www.google.com/url?sa=D&q=http://groups.google.com/group/python-shef…> Thanks, Daley Chetwynd

1 0

Next CamPUG meeting: Tue 6th Dec 2011
by Tony Ibbs Nov. 26, 2011

Nov. 26, 2011

As posted to our google group: The next meeting should be a coding meeting, 7.30pm at RealVNC (http://tinyurl.com/realvncoffices). I imagine we may continue thinking about the code we worked on last time: http://groups.google.com/group/campug/msg/f63ffa366a64dd09 Meetings after that should be: * Tuesday 10th January, talks (note that this is the SECOND Tuesday) * Tuesday 7th February, dojo/coding * Tuesday 6th March, talks again Tibs

1 0

Python problem with ArcGIS
by Daniel Tuck Nov. 25, 2011

Nov. 25, 2011

http://www.esri.com/news/arcuser/0111/charmsnake.html ListLayers returns a Python list object containing layer objects. The str function converts a value (e.g., integer) to a string. Hi everyone, I'm trying to adapt the script on this page to return the workspace of every layer within an ArcGIS MXD… I think the stumbling block I'm up to is that I have a python list object and from there I'm having problems getting to the properties of the indivual layers. This is the code I have so far.. very messy in the middle! import os import sys import arcpy import arcpy.mapping as mapping # get input parameters outDir = arcpy.GetParameterAsText(0) packageMap = arcpy.GetParameter(1) checkBrokenLayers = arcpy.GetParameter(2) mxd_path = arcpy.GetParameter(3) # get the map document in which this code is running ("Current" keyword) mxd = mapping.MapDocument(mxd_path) # build a pathname to the output report (text file) reportPath = outDir + '\\' + 'test.txt' # open the file (will be created if it doesn't exist) reportFile = open(reportPath, 'w') arcpy.AddMessage('Writing report to ' + reportPath) # start writing report text to a string variable reportText = 'Title: ' + mxd.title + '\n' reportText += 'Author: ' + mxd.author + '\n' reportText += 'Description: ' + mxd.description + '\n' reportText += '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~' + '\n' # if the user chose to do so, create map package if packageMap: packagePath = outDir + '\\' + mxd.title.replace('.', '_') + '.mpk' if (os.path.exists(packagePath)): arcpy.AddMessage('Map package already exists (' + packagePath + ')') else: arcpy.AddMessage('Creating map package (' + packagePath + ')' ) arcpy.PackageMap_management(mxd.filePath, packagePath) # loop thru all data frames in the map dataFrames = mapping.ListDataFrames(mxd, '') for frame in dataFrames: # report data frame name and spatial reference reportText += '\nData Frame: ' + frame.name + '\n' reportText += 'Spatial Reference: ' + frame.spatialReference.name + '\n' # get all layers in this data frame # "*" '' layers = mapping.ListLayers(mxd, '', frame) i = 0 # layer index position # loop thru all layers in the data frame for lyr in layers: # report index position and name - ' + lyr.longname + # so a python reference to the layer has been created # http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Layer/00s3000… # now need to call it??? # here a list is being returned rather than the layer object itself? objLayer = arcpy.mapping.Layer(lyr) # i.e get object from layer reference LYR reportText += '\tLayer ' + str(i) + ': ' + lyr.name + ':' # + objLayer.workspace + '\n' i += 1 # same as i = i + 1 # if the user has requested it, check for layers with a missing data source if checkBrokenLayers: arcpy.AddMessage('Checking for missing data sources') brokenList = mapping.ListBrokenDataSources(mxd) # report the count of broken layers reportText += '\nFound ' + str(len(brokenList)) + ' layers with missing data.' # loop thru all broken layers in the list for broken in brokenList: # report broken layer name reportText += '\t- ' + broken.name + '\n' # write the text stored in the reportText variable to the output file reportFile.write(reportText) reportFile.close() # close the file del mxd # delete the mxd object Any help appreciated! Dan,

1 0

Dojo / workshop on "TDD Django with Selenium" - any interest?
by Harry Percival Nov. 16, 2011

Nov. 16, 2011

Hi-ho python peeps, Would anyone be interested in a dojo / worksop on the topic of test-driven Django development, with Selenium? I've been working on a tutorial on the topic (< https://github.com/hjwp/Test-Driven-Django-Tutorial> *) . I'm not claiming to be a massive expert, but it's what I've learned at work over the last year or so, so it's fresh in my mind... I'm pretty sure I could get a couple of (ex?) colleagues to help present... Would be aimed at beginners / people who don't know Selenium / people who want to learn Django the "right" way / people who want to learn TDD... If you already know Django and Selenium back to front, it would probably be of less interest, although there may be some interesting discussions around integrating the Django test runner, WebDriver vs Selenium-RC etc... So, trying to get an idea of numbers - would anyone be interested? London area, venue suggestions also gratefully accepted... HP * work in progress! for example, the .jar file really isn't necessary... -- ------------------------------ Harry J.W. Percival ------------------------------ @hjwp Mobile: +44 (0) 78877 02511 Skype: harry.percival

14 20

Re: [python-uk] memoize & ordering of kwargs.items()
by Jonathan Nov. 11, 2011

Nov. 11, 2011

Thanks Ross. The thing is, the 'kwargs' dict that callers pass in is not the same dict as the kwargs that my wrapper function receives. My 'wrapper' function always receives a brand-new dictionary, created from scratch as the function is invoked. Hence, the kwargs never has any 'history' - they are always newly created, from the keyword args passed in to the function. This, I suspect, is why their 'items()' always appear in the same order. I tried calling 'wrapper' with keyword args in a different order: wrapper(x=1, y=2, z=3) wrapper(y=2, x=1, z=3) ...etc, for every permutation and by passing in **kwargs dictionaries with different histories (detailed in my original mail): wrapper(**kwargs) but in every case, inside wrapper... def wrapper(*args, **kwargs): print kwargs.items() ...the items call always returns the dictionary's content in the same order. So I *suspect*, but cannot yet prove, that the '**' mechanism is already sorting the keyword args used to construct the kwargs dictionary (or is doing some equivalent, such that the dicts always end up identical-including-order-of-items()) I agree that 'to be sure' is enough of a justification for including the 'sorted' call anyway. But it irks me that I can't write a test to show it. Jonathan On 11/11/2011 08:50, Ross Lawley wrote: > Hi, > > From the docs: http://docs.python.org/library/stdtypes.html#dict.items > > "CPython implementation detail: Keys and values are listed in an > arbitrary order which is non-random, varies across Python > implementations, and depends on the dictionary's history of insertions > and deletions." > > Given that I would use sorted to guarantee order. Then no matter what > anyone has done to the kwargs passed in the memoizing will not have a bug. > > Ross > > On Fri, Nov 11, 2011 at 8:42 AM, Jonathan <tartley(a)tartley.com > <mailto:tartley@tartley.com>> wrote: > > Hey, > > I've been writing my own 'memoize' function a lot recently - I'm > using it as an interview question. > > Here's what I've got (with a corresponding series of tests): > > def memoize(wrapped): > > cache = {} > > @wraps(wrapped) > def wrapper(*args, **kwargs): > key = (args, tuple(kwargs.items())) > if key not in cache: > cache[key] = wrapped(*args, **kwargs) > return cache[key] > > return wrapper > > Yesterday a candidate pointed out that this is wrong because > .items() for two equal dictionaries might return the (key,value) > pairs in a different order, presumably dependent on the respective > dictionary's history. This would produce a different 'key' and > hence an erroneous cache miss. > > This implies that the second line of 'wrapper' should become: > > key = (args, tuple(sorted(kwargs.items()))) > > (I've added 'sorted') > Looking at lru_cache, I see that is exactly how it is implemented > there. > http://hg.python.org/cpython/file/default/Lib/functools.py > > However, I'm unable to come up with a test that proves this is > necessary. I'm can create two equal dictionaries which return > their .items() in a different order: > > # The intent is that 'small.items()' comes out in a > different order > # than 'large.items()' > small = {'x':1, 'y':5} > large = {hex(i): i for i in range(257)} > large.update(small) > for i in range(257): > del large[hex(i)] > > >>> print small.items() > [('y', 5), ('x', 1)] > >>> print large.items() > [('x', 1), ('y', 5)] > > If I could magically transfer these dictionaries directly into the > 'kwargs' of wrapper, then I think I'd be done. However, I have to > pass these dictionaries in to wrapper via the '**' mechanism. > Printing 'kwargs.items()' within 'wrapper' shows that equal > dictionaries always return their items in the same order, so the > 'sorted' call is apparently not necessary. > > Is 'sorted' really required? If so, how can I write a test to > demonstrate it? > > Best regards, > > Jonathan > > -- > Jonathan Hartley tartley(a)tartley.com <mailto:tartley@tartley.com> > http://tartley.com > Made of meat. +44 7737 062 225 <tel:%2B44%207737%20062%20225> > twitter/skype: tartley > > > _______________________________________________ > python-uk mailing list > python-uk(a)python.org <mailto:python-uk@python.org> > http://mail.python.org/mailman/listinfo/python-uk > > > > > _______________________________________________ > python-uk mailing list > python-uk(a)python.org > http://mail.python.org/mailman/listinfo/python-uk -- Jonathan Hartley tartley(a)tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley

3 2

Re: [python-uk] memoize & ordering of kwargs.items()
by Jonathan Nov. 11, 2011

Nov. 11, 2011

On 11/11/2011 09:34, René Dudfield wrote: > On Fri, Nov 11, 2011 at 10:23 AM, Jonathan <tartley(a)tartley.com > <mailto:tartley@tartley.com>> wrote: > > That's good to know René, but I *think* it's orthogonal to the > question. Please correct me if I'm wrong. > > If PyPy returns items in a different order than CPython, that > doesn't matter to me, so long as every invocation of my function > in a particular process ends up receiving a particular order, > that doesn't change until the process ends (or the cache is cleared.) > > The dict subclass is a great idea - but if the change in order > isn't ever manifested by a regular dict, then it implies to me > that the 'sorted' call isn't actually required in real life, so > the need for this whole test disappears. > > See also my imminent reply to Ross. > > Jonathan > > > > Just because your code works Now, does not mean it will work in the > future. Especially when the ordering is explicitly marked as a > non-deterministic implementation detail. Relying on it to be > deterministic when it is stated to be non-deterministic will mean that > in the future you could get a failure. > > Relying on implementation quirks is fine, but not reliable. In your > case though, the worst that can happen now is a cache miss. In the > future though, your cache could be completely useless because every > time the data could be in a different order. > > Testing on multiple platforms allows you to sometimes see a failure > condition, which lets you test it. Likewise, using a randomised dict > subclass will let you test your code as if there is an error condition. > > cya. > > ps. my randomised items subclass is buggy... I forgot that > random.shuffle works in place, and doesn't return a shuffled sequence. > > > _______________________________________________ > python-uk mailing list > python-uk(a)python.org > http://mail.python.org/mailman/listinfo/python-uk Hey René, Thanks for that! I guess I was getting distracted because I was interpreting the situation as the fact that .items() for an *arbitrary* dictionary may well be non-deterministic, but in this particular case, for dictionaries newly-created by the '**' mechanism, they appear to be non-deterministic. I was forgetting that this is (presumably) also an implementation detail. Regardless, it sounds like you are probably right on all counts. Thanks for your enlightening thoughts. Cheers, Jonathan -- Jonathan Hartley tartley(a)tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley

2 1

Re: [python-uk] memoize & ordering of kwargs.items()
by Jonathan Nov. 11, 2011

Nov. 11, 2011

On 11/11/2011 09:35, Jonathan wrote: > On 11/11/2011 09:24, Duncan Booth wrote: >> pick keys that that hash to the same value modulo the size of the >> dictionary. Since the dictionary copy copies the keys in the order >> they are stored you will get the same hash conflict in the copy as >> the original. > > Brilliant, thank-you. So I was just being 'lucky' in my choices of > dict keys. I believe this is the answer I am looking for. René & > Ross's point about being mindful of different implementations is still > very pertinent in my mind, but I'm happy for now. > > Thanks everyone! > > Jonathan > For completeness, my final test is therefore: def test_not_dependant_on_order_of_kwargs(self): calls = [] @memoize def counter(**kwargs): calls.append(1) for first in ascii_letters: for second in ascii_letters: forward = {first:0, second:0} reverse = {second:0, first:0} del calls[:] counter(**forward) counter(**reverse) self.assertEqual(len(calls), 1, '%d for %s,%s' % (len(calls), first, second)) Without the 'sorted', this fails with: ====================================================================== FAIL: test_not_dependant_on_order_of_kwargs (__main__.MemoizeTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "./test.py", line 58, in test_not_dependant_on_order_of_kwargs '%d for %s,%s' % (len(calls), first, second)) AssertionError: 2 for a,i ---------------------------------------------------------------------- Jubilant! Jonathan -- Jonathan Hartley tartley(a)tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley

1 1

Re: [python-uk] memoize & ordering of kwargs.items()
by Jonathan Nov. 11, 2011

Nov. 11, 2011

On 11/11/2011 09:24, Duncan Booth wrote: > pick keys that that hash to the same value modulo the size of the > dictionary. Since the dictionary copy copies the keys in the order > they are stored you will get the same hash conflict in the copy as the > original. Brilliant, thank-you. So I was just being 'lucky' in my choices of dict keys. I believe this is the answer I am looking for. René & Ross's point about being mindful of different implementations is still very pertinent in my mind, but I'm happy for now. Thanks everyone! Jonathan -- Jonathan Hartley tartley(a)tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley

1 0

Re: [python-uk] memoize & ordering of kwargs.items()
by Jonathan Nov. 11, 2011

Nov. 11, 2011

That's good to know René, but I *think* it's orthogonal to the question. Please correct me if I'm wrong. If PyPy returns items in a different order than CPython, that doesn't matter to me, so long as every invocation of my function in a particular process ends up receiving a particular order, that doesn't change until the process ends (or the cache is cleared.) The dict subclass is a great idea - but if the change in order isn't ever manifested by a regular dict, then it implies to me that the 'sorted' call isn't actually required in real life, so the need for this whole test disappears. See also my imminent reply to Ross. Jonathan On 11/11/2011 08:49, René Dudfield wrote: > Good morning, > > 1. Run it on different versions of python, or on different > machines/OSes. That usually results in different dictionary ordering. > 2. create a dict subclass that returns items always randomised. > > class my_fun_randomising_item_dict_subclass(dict): > def items(self): > return random.shuffle(dict.items(self, key)) > def make_key(kwargs): > return (args, > tuple(sorted(my_fun_randomising_item_dict_subclass(kwargs).items()))) > > cheers, > > On Fri, Nov 11, 2011 at 9:42 AM, Jonathan <tartley(a)tartley.com > <mailto:tartley@tartley.com>> wrote: > > Hey, > > I've been writing my own 'memoize' function a lot recently - I'm > using it as an interview question. > > Here's what I've got (with a corresponding series of tests): > > def memoize(wrapped): > > cache = {} > > @wraps(wrapped) > def wrapper(*args, **kwargs): > key = (args, tuple(kwargs.items())) > if key not in cache: > cache[key] = wrapped(*args, **kwargs) > return cache[key] > > return wrapper > > Yesterday a candidate pointed out that this is wrong because > .items() for two equal dictionaries might return the (key,value) > pairs in a different order, presumably dependent on the respective > dictionary's history. This would produce a different 'key' and > hence an erroneous cache miss. > > This implies that the second line of 'wrapper' should become: > > key = (args, tuple(sorted(kwargs.items()))) > > (I've added 'sorted') > Looking at lru_cache, I see that is exactly how it is implemented > there. > http://hg.python.org/cpython/file/default/Lib/functools.py > > However, I'm unable to come up with a test that proves this is > necessary. I'm can create two equal dictionaries which return > their .items() in a different order: > > # The intent is that 'small.items()' comes out in a > different order > # than 'large.items()' > small = {'x':1, 'y':5} > large = {hex(i): i for i in range(257)} > large.update(small) > for i in range(257): > del large[hex(i)] > > >>> print small.items() > [('y', 5), ('x', 1)] > >>> print large.items() > [('x', 1), ('y', 5)] > > If I could magically transfer these dictionaries directly into the > 'kwargs' of wrapper, then I think I'd be done. However, I have to > pass these dictionaries in to wrapper via the '**' mechanism. > Printing 'kwargs.items()' within 'wrapper' shows that equal > dictionaries always return their items in the same order, so the > 'sorted' call is apparently not necessary. > > Is 'sorted' really required? If so, how can I write a test to > demonstrate it? > > Best regards, > > Jonathan > > -- > Jonathan Hartley tartley(a)tartley.com <mailto:tartley@tartley.com> > http://tartley.com > Made of meat. +44 7737 062 225 <tel:%2B44%207737%20062%20225> > twitter/skype: tartley > > > _______________________________________________ > python-uk mailing list > python-uk(a)python.org <mailto:python-uk@python.org> > http://mail.python.org/mailman/listinfo/python-uk > > > > > _______________________________________________ > python-uk mailing list > python-uk(a)python.org > http://mail.python.org/mailman/listinfo/python-uk -- Jonathan Hartley tartley(a)tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley

2 1