A new dictionary implementation

Hi, Now that issue 13703 has been largely settled, I want to propose my new dictionary implementation again. It is a little more polished than before. https://bitbucket.org/markshannon/hotpy_new_dict Object-oriented benchmarks use considerably less memory and are sometimes faster (by a small amount). (I've only benchmarked on my old 32bit machine) E.g 2to3 No speed change -28% memory GCbench +10% speed -47% memory Other benchmarks show little or no change in behaviour, mainly minor memory savings. If an application is OO and uses lots of memory the new dict will save a lot of memory and maybe boost performance. Other applications will be largely unaffected. It passes all the tests. (I had to change a couple that relied on dict repr() ordering) Cheers, Mark.

Hi, On Sun, 29 Jan 2012 10:31:48 +0000 Mark Shannon <mark@hotpy.org> wrote:
I briefly took a look at your code yesterday and it looked generally reasonable to me. It would be nice to open an issue on http://bugs.python.org so that we can review it there (just fill the "repository" field and use the "create patch" button). Regards Antoine.

2012/1/29 Mark Shannon <mark@hotpy.org>:
If you're serious about changing the dictionary implementation, I think you should write a PEP. It should explain the new dicts advantages (and disadvantages?) and give comprehensive benchmark numbers. Something along the lines of http://www.python.org/dev/peps/pep-3128/ I should think. -- Regards, Benjamin

On Sun, 29 Jan 2012 09:56:11 -0500 Benjamin Peterson <benjamin@python.org> wrote:
"New dictionary implementation" is a misnomer here. Mark's patch merely allows to share the keys array between several dictionaries. The lookup algorithm remains exactly the same as far as I've read. It's actually much less invasive than e.g. Martin's AVL trees-for-hash-collisions proposal. Regards Antoine.

Mark Shannon wrote:
I don't quite follow how that could work. If I have this: class C: pass a = C() b = C() a.spam = 1 b.ham = 2 how can a.__dict__ and b.__dict__ share key arrays? I've tried reading the source, but I'm afraid I don't understand it well enough to make sense of it. -- Steven

On 30/01/12 00:30:14, Steven D'Aprano wrote:
They can't. But then, your class is atypical. Usually, classes initialize all the attributes of their instances in the __init__ method, perhaps like so: class D: def __init__(self, ham=None, spam=None): self.ham = ham self.spam = spam As long as you follow the common practice of not adding any attributes after the object has been initialized, your instances can share their keys array. Mark's patch will do that. You'll still be allowed to have different attributes per instance, but if you do that, then the patch doesn't buy you much. -- HansM

On Wed, Feb 1, 2012 at 9:13 AM, Hans Mulder <hansmu@xs4all.nl> wrote:
Hey, I like this! It's a subtle encouragement for developers to initialize all their instance variables in their __init__ or __new__ method, with a (modest) performance improvement for a carrot. (Though I have to admit I have no idea how you do it. Wouldn't the set of dict keys be different while __init__ is in the middle of setting the instance variables?) Another question: a common pattern is to use (immutable) class variables as default values for instance variables, and only set the instance variables once they need to be different. Does such a class benefit from your improvement?
-- HansM
-- --Guido van Rossum (python.org/~guido)

Guido van Rossum <guido <at> python.org> writes:
While I absolutely cannot speak to this implementation. Traditionally this type of approach is refered to as maps, and was pioneered in SELF, originally presented at OOPSLA '89: http://dl.acm.org/citation.cfm?id=74884 . PyPy also uses these maps to back it's object, although from what I've read the implementation looks nothing like the proposed one for CPython, you can read about that here: http://bit.ly/zwlOkV , and if you're really excited about this you can read our implementation here: https://bitbucket.org/pypy/pypy/src/default/pypy/objspace/std/mapdict.py . Alex

Alex <alex.gaynor <at> gmail.com> writes:
about that here: http://bit.ly/zwlOkV , and if you're really excited about this
Err, here's a working version of the bit.ly link: http://bit.ly/a05h6r Alex

Just a quick update. I've been analysing and profile the behaviour of my new dict and messing about with various implementation options. I've settled on a new implementation. Its the same basic idea, but with better locality of reference for unshared keys. Guido asked:
For those instances which keep the default, yes. Otherwise the answer is, as Martin pointed out, it could yes provided that adding a new key does not force a resize. Although it is a bit arbitrary when a resize occurs. The new version will incorporate this behaviour. Expect version 2 soon. Cheers, Mark.

Hi, Version 2 is now available. Version 2 makes as few changes to tunable constants as possible, and generally does not change iteration order (so repr() is unchanged). All tests pass (the only changes to tests are for sys.getsizeof() ). Repository: https://bitbucket.org/markshannon/cpython_new_dict Issue http://bugs.python.org/issue13903 Performance changes are basically zero for non-OO code. Average -0.5% speed change on 2n3 benchamrks, a few benchmarks show a small reduction in memory use. (see notes below) GCbench uses 47% less memory and is 12% faster. 2to3, which seems to be the only "realistic" benchmark that runs on Py3, shows no change in speed and uses 10% less memory. All benchmarks and tests performed on old, slow 32bit machine with linux. Do please try it on your machine(s). If accepted, the new dict implementation will allow a useful optimisation of the LOAD_GLOBAL (and possibly LOAD_ATTR) bytecode: By testing to see if the (immutable) keys-tables is the expected table, the value can accessed directly by index, rather than by name. Cheers, Mark. Notes: All benchmarks from http://hg.python.org/benchmarks/ using the -m flag to get memory usage data. I've ignored the json benchmarks which shows unstable behaviour on my machine. Tiny changes to the dict being serialized or to the random seed can change the relative speed of my implementation vs CPython from -25% to +10%.

Hi Mark, I've just cloned :
Repository: https://bitbucket.org/markshannon/cpython_new_dict
Do please try it on your machine(s).
.... that's a: Linux random 3.1.0-1-amd64 #1 SMP Tue Jan 10 05:01:58 UTC 2012 x86_64 GNU/Linux and I'm getting: gcc -pthread -c -Wno-unused-result -g -O0 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o Objects/dictobject.o Objects/dictobject.c gcc -pthread -c -Wno-unused-result -g -O0 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o Objects/memoryobject.o Objects/memoryobject.c Objects/dictobject.c: In function ‘dict_popitem’: Objects/dictobject.c:2208:5: error: ‘PyDictKeyEntry’ has no member named ‘me_value’ make: *** [Objects/dictobject.o] Error 1 make: *** Waiting for unfinished jobs.... Cheers francis

Hi Mark,
I've tried to run the test suite but I'm getting a SyntaxError: (may be you know it's just the first time that I try the tool): ============================= ci@random:~/prog/cpython/benchmarks$ python perf.py -r -b apps python ../cpython_new_dict/python Running 2to3... INFO:root:Running ../cpython_new_dict/python lib/2to3/2to3 -f all lib/2to3_data Traceback (most recent call last): File "perf.py", line 2236, in <module> main(sys.argv[1:]) File "perf.py", line 2192, in main options))) File "perf.py", line 1279, in BM_2to3 return SimpleBenchmark(Measure2to3, *args, **kwargs) File "perf.py", line 706, in SimpleBenchmark *args, **kwargs) File "perf.py", line 1275, in Measure2to3 return MeasureCommand(command, trials, env, options.track_memory) File "perf.py", line 1223, in MeasureCommand CallAndCaptureOutput(command, env=env) File "perf.py", line 1053, in CallAndCaptureOutput raise RuntimeError(u"Benchmark died: " + unicode(stderr, 'ascii')) RuntimeError: Benchmark died: Traceback (most recent call last): File "lib/2to3/2to3", line 3, in <module> from lib2to3.main import main File "/home/ci/prog/cpython/benchmarks/lib/2to3/lib2to3/main.py", line 47 except os.error, err: ^ SyntaxError: invalid syntax ============================= And the baseline is: Python 2.7.2+ (but it also gives me an SyntaxError running on python3 default (e50db1b7ad7b) What I'm doing wrong ? (from it's doc: “This project is intended to be an authoritative source of benchmarks for all Python implementations.”) Thanks in advance ! francis

On 08/02/2012 15:16, Mark Shannon wrote:
In your first version 2to3 used 28% less memory. Do you know why it's worse in this version? Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Hi All, My new dictionary implementation has been further updated in light of various comments and questions. The code is here: https://bitbucket.org/markshannon/cpython_new_dict The resizing code has been streamlined. This means that resizing by doubling does not suffer any real performance penalty versus quadrupling. Doubling uses less memory for most benchmarks (it never uses more memory). Michael Foord wrote:
The answer is that the second version used quadrupling rather than doubling for resizing. All tests pass. test_sys has been altered to allow for the different size of the dictionary. One test in test_pprint has been disabled. This test is broken anyway, see http://bugs.python.org/issue13907. In general, for the new dictionary implementation, with doubling at a resize, speed is unchanged and memory usage is reduced. On "average": ~1% slow down, ~10% reduction in memory use. Full set of benchmarks for new dict with doubling and quadrupling attached. Unfortunately the benchmarking program introduces systematic errors for timings, but it's the best we have at the moment. Note that the json benchmark is unstable and should be ignored. The GC benchmark might be unstable as well, I haven't experimented. The memory usage numbers seems to be more reliable. Revised PEP to follow. Cheers, Mark. Report on Linux vespasian 2.6.24-30-generic #1 SMP i686 Total CPU cores: 1 New dict (doubling) vs. CPython =============================== Speed ----- ### call_simple ### Min: 1.131625 -> 1.216112: 1.07x slower Avg: 1.143615 -> 1.223153: 1.07x slower Significant (t=-95.62) Stddev: 0.00694 -> 0.00745: 1.0734x larger Timeline: b'http://tinyurl.com/7ey5unc' ### fastpickle ### Min: 2.365578 -> 2.315522: 1.02x faster Avg: 2.379824 -> 2.326625: 1.02x faster Significant (t=28.84) Stddev: 0.00984 -> 0.00857: 1.1479x smaller Timeline: b'http://tinyurl.com/75nk6fg' ### fastunpickle ### Min: 2.043093 -> 2.083532: 1.02x slower Avg: 2.056331 -> 2.103432: 1.02x slower Significant (t=-31.72) Stddev: 0.00667 -> 0.00811: 1.2144x larger Timeline: b'http://tinyurl.com/7r24oyr' ### float ### Min: 0.256634 -> 0.242119: 1.06x faster Avg: 0.281768 -> 0.252510: 1.12x faster Significant (t=34.46) Stddev: 0.00985 -> 0.00911: 1.0811x smaller Timeline: b'http://tinyurl.com/72cy3ap' ### formatted_logging ### Min: 1.801701 -> 1.886665: 1.05x slower Avg: 1.825188 -> 1.907127: 1.04x slower Significant (t=-31.16) Stddev: 0.01431 -> 0.01188: 1.2046x smaller Timeline: b'http://tinyurl.com/7yjstcs' ### iterative_count ### Min: 0.531924 -> 0.570944: 1.07x slower Avg: 0.549278 -> 0.586633: 1.07x slower Significant (t=-17.09) Stddev: 0.01287 -> 0.00856: 1.5042x smaller Timeline: b'http://tinyurl.com/7afe77x' ### json_dump ### Min: 1.807933 -> 2.361769: 1.31x slower Avg: 1.833581 -> 2.375774: 1.30x slower Significant (t=-249.39) Stddev: 0.01165 -> 0.01003: 1.1623x smaller Timeline: b'http://tinyurl.com/7nkvfdf' ### json_load ### Min: 1.494502 -> 1.733510: 1.16x slower Avg: 1.513454 -> 1.780639: 1.18x slower Significant (t=-50.91) Stddev: 0.01285 -> 0.03481: 2.7088x larger Timeline: b'http://tinyurl.com/78uex2v' ### nbody ### Min: 1.195176 -> 1.111036: 1.08x faster Avg: 1.205988 -> 1.117919: 1.08x faster Significant (t=64.16) Stddev: 0.00770 -> 0.00591: 1.3028x smaller Timeline: b'http://tinyurl.com/88ylc46' ### nqueens ### Min: 1.205363 -> 1.270263: 1.05x slower Avg: 1.219677 -> 1.286260: 1.05x slower Significant (t=-34.55) Stddev: 0.00992 -> 0.00934: 1.0626x smaller Timeline: b'http://tinyurl.com/6nbjq6y' ### regex_compile ### Min: 2.139500 -> 2.410691: 1.13x slower Avg: 2.150651 -> 2.429258: 1.13x slower Significant (t=-168.69) Stddev: 0.00736 -> 0.00907: 1.2333x larger Timeline: b'http://tinyurl.com/88xr386' ### regex_effbot ### Min: 0.194415 -> 0.200979: 1.03x slower Avg: 0.199182 -> 0.206103: 1.03x slower Significant (t=-4.83) Stddev: 0.00715 -> 0.00717: 1.0028x larger Timeline: b'http://tinyurl.com/6ll7pdv' ### regex_v8 ### Min: 0.233819 -> 0.219033: 1.07x faster Avg: 0.249771 -> 0.227338: 1.10x faster Significant (t=7.69) Stddev: 0.01553 -> 0.01358: 1.1440x smaller Timeline: b'http://tinyurl.com/74dqdsy' ### richards ### Min: 0.698549 -> 0.727000: 1.04x slower Avg: 0.717596 -> 0.750720: 1.05x slower Significant (t=-16.90) Stddev: 0.00896 -> 0.01058: 1.1812x larger Timeline: b'http://tinyurl.com/76cvgyj' ### silent_logging ### Min: 0.282622 -> 0.302598: 1.07x slower Avg: 0.291308 -> 0.313256: 1.08x slower Significant (t=-13.24) Stddev: 0.00791 -> 0.00865: 1.0937x larger Timeline: b'http://tinyurl.com/6vovnav' ### simple_logging ### Min: 1.622786 -> 1.796359: 1.11x slower Avg: 1.660321 -> 1.821784: 1.10x slower Significant (t=-38.86) Stddev: 0.02772 -> 0.00974: 2.8450x smaller Timeline: b'http://tinyurl.com/777y27t' ### threaded_count ### Min: 0.501266 -> 0.532309: 1.06x slower Avg: 0.521384 -> 0.549346: 1.05x slower Significant (t=-13.74) Stddev: 0.01074 -> 0.00957: 1.1228x smaller Timeline: b'http://tinyurl.com/889qegb' ### unpack_sequence ### Min: 0.000212 -> 0.000212: no change Avg: 0.000232 -> 0.000242: 1.04x slower Significant (t=-6.07) Stddev: 0.00026 -> 0.00026: 1.0096x larger Timeline: b'http://tinyurl.com/8yzsnmo' ### 2to3 ### 36.702294 -> 37.234327: 1.01x slower ### mako ### Min: 0.982504 -> 0.887246: 1.11x faster Avg: 1.008269 -> 0.934528: 1.08x faster Significant (t=57.62) Stddev: 0.01178 -> 0.01645: 1.3970x larger Timeline: b'http://tinyurl.com/7p2j9s2' ### mako ### Min: 0.753596 -> 0.874137: 1.16x slower Avg: 0.793711 -> 0.915435: 1.15x slower Significant (t=-85.92) Stddev: 0.01607 -> 0.01561: 1.0293x smaller Timeline: b'http://tinyurl.com/8xu5nh9' The following not significant results are hidden, use -v to show them: call_method, call_method_slots, call_method_unknown, normal_startup, pidigits, startup_nosite. Memory ------ ### call_method ### Mem max: 4164.000 -> 4032.000: 1.0327x smaller Usage over time: b'http://tinyurl.com/7saraes' ### call_method_slots ### Mem max: 4164.000 -> 4028.000: 1.0338x smaller Usage over time: b'http://tinyurl.com/722jr7s' ### call_method_unknown ### Mem max: 4464.000 -> 4068.000: 1.0973x smaller Usage over time: b'http://tinyurl.com/6oayz7q' ### call_simple ### Mem max: 4148.000 -> 4008.000: 1.0349x smaller Usage over time: b'http://tinyurl.com/75v3szx' ### fastpickle ### Mem max: 5108.000 -> 4876.000: 1.0476x smaller Usage over time: b'http://tinyurl.com/6s3q7pt' ### fastunpickle ### Mem max: 5132.000 -> 4896.000: 1.0482x smaller Usage over time: b'http://tinyurl.com/7fjhljq' ### float ### Mem max: 8592.000 -> 6772.000: 1.2688x smaller Usage over time: b'http://tinyurl.com/7ke4acp' ### formatted_logging ### Mem max: 10732.000 -> 10892.000: 1.0149x larger Usage over time: b'http://tinyurl.com/83z4f2a' ### iterative_count ### Mem max: 4200.000 -> 4068.000: 1.0324x smaller Usage over time: b'http://tinyurl.com/77ph38p' ### json_dump ### Mem max: 4944.000 -> 4600.000: 1.0748x smaller Usage over time: b'http://tinyurl.com/896y5he' ### json_load ### Mem max: 4940.000 -> 4596.000: 1.0748x smaller Usage over time: b'http://tinyurl.com/77mw6wz' ### nbody ### Mem max: 4172.000 -> 4024.000: 1.0368x smaller Usage over time: b'http://tinyurl.com/7h2795d' ### normal_startup ### Mem max: 3772.000 -> 3632.000: 1.0385x smaller Usage over time: b'http://tinyurl.com/84nwxz7' ### nqueens ### Mem max: 4272.000 -> 4132.000: 1.0339x smaller Usage over time: b'http://tinyurl.com/7sshb2v' ### pidigits ### Mem max: 4248.000 -> 4108.000: 1.0341x smaller Usage over time: b'http://tinyurl.com/82urnw2' ### regex_compile ### Mem max: 8572.000 -> 8904.000: 1.0387x larger Usage over time: b'http://tinyurl.com/6nukmpu' ### regex_effbot ### Mem max: 4864.000 -> 4736.000: 1.0270x smaller Usage over time: b'http://tinyurl.com/767ge37' ### regex_v8 ### Mem max: 7508.000 -> 7860.000: 1.0469x larger Usage over time: b'http://tinyurl.com/72lxz75' ### richards ### Mem max: 4464.000 -> 4356.000: 1.0248x smaller Usage over time: b'http://tinyurl.com/7794rv7' ### silent_logging ### Mem max: 4844.000 -> 4496.000: 1.0774x smaller Usage over time: b'http://tinyurl.com/8332w2a' ### simple_logging ### Mem max: 6952.000 -> 5352.000: 1.2990x smaller Usage over time: b'http://tinyurl.com/8yk7svt' ### startup_nosite ### Mem max: 2872.000 -> 2748.000: 1.0451x smaller Usage over time: b'http://tinyurl.com/72bbdnp' ### threaded_count ### Mem max: 4232.000 -> 4084.000: 1.0362x smaller Usage over time: b'http://tinyurl.com/8yrshqh' ### unpack_sequence ### Mem max: 5688.000 -> 5416.000: 1.0502x smaller Usage over time: b'http://tinyurl.com/7jy6dpl' ### mako ### Mem max: 9012.000 -> 8580.000: 1.0503x smaller Usage over time: b'http://tinyurl.com/82qzfvo' ### 2to3 ### Mem max: 32684.000 -> 22356.000: 1.4620x smaller Usage over time: b'http://tinyurl.com/77hbok6' ### gc ### Mem max: 92600.000 -> 49000.000: 1.8898x smaller Usage over time: b'http://tinyurl.com/86vswmt' New dict (quadrupling) vs. CPython ================================== Speed ----- ### call_simple ### Min: 1.217095 -> 1.269468: 1.04x slower Avg: 1.228556 -> 1.282715: 1.04x slower Significant (t=-55.48) Stddev: 0.00826 -> 0.00864: 1.0459x larger Timeline: b'http://tinyurl.com/7qtz7hk' ### fastpickle ### Min: 2.421091 -> 2.333122: 1.04x faster Avg: 2.447303 -> 2.348845: 1.04x faster Significant (t=41.67) Stddev: 0.01290 -> 0.01062: 1.2142x smaller Timeline: b'http://tinyurl.com/8634dn3' ### fastunpickle ### Min: 2.069723 -> 2.024058: 1.02x faster Avg: 2.082294 -> 2.039120: 1.02x faster Significant (t=17.92) Stddev: 0.00775 -> 0.01517: 1.9572x larger Timeline: b'http://tinyurl.com/73t6ldd' ### formatted_logging ### Min: 1.839560 -> 1.950483: 1.06x slower Avg: 1.868685 -> 1.973164: 1.06x slower Significant (t=-36.79) Stddev: 0.01822 -> 0.00843: 2.1607x smaller Timeline: b'http://tinyurl.com/6pej9zy' ### iterative_count ### Min: 0.482927 -> 0.564538: 1.17x slower Avg: 0.496796 -> 0.575018: 1.16x slower Significant (t=-41.67) Stddev: 0.00982 -> 0.00893: 1.1002x smaller Timeline: b'http://tinyurl.com/735skdg' ### json_dump ### Min: 1.863940 -> 2.130464: 1.14x slower Avg: 1.905174 -> 2.158846: 1.13x slower Significant (t=-44.83) Stddev: 0.03699 -> 0.01525: 2.4253x smaller Timeline: b'http://tinyurl.com/8yx7oso' ### json_load ### Min: 1.586767 -> 1.761993: 1.11x slower Avg: 1.610569 -> 1.793440: 1.11x slower Significant (t=-80.38) Stddev: 0.01279 -> 0.00976: 1.3097x smaller Timeline: b'http://tinyurl.com/6ny4qr5' ### regex_compile ### Min: 2.163662 -> 2.332177: 1.08x slower Avg: 2.174792 -> 2.349151: 1.08x slower Significant (t=-89.97) Stddev: 0.00824 -> 0.01095: 1.3287x larger Timeline: b'http://tinyurl.com/7htwu63' ### regex_effbot ### Min: 0.203670 -> 0.197273: 1.03x faster Avg: 0.208266 -> 0.202605: 1.03x faster Significant (t=3.74) Stddev: 0.00752 -> 0.00763: 1.0143x larger Timeline: b'http://tinyurl.com/7znmo3r' ### regex_v8 ### Min: 0.226723 -> 0.221915: 1.02x faster Avg: 0.239020 -> 0.230629: 1.04x faster Significant (t=3.22) Stddev: 0.01280 -> 0.01325: 1.0349x larger Timeline: b'http://tinyurl.com/7m99e3s' ### richards ### Min: 0.708626 -> 0.727219: 1.03x slower Avg: 0.729989 -> 0.747773: 1.02x slower Significant (t=-8.71) Stddev: 0.01022 -> 0.01020: 1.0020x smaller Timeline: b'http://tinyurl.com/6ueszbh' ### silent_logging ### Min: 0.272886 -> 0.306478: 1.12x slower Avg: 0.281103 -> 0.314645: 1.12x slower Significant (t=-19.17) Stddev: 0.00832 -> 0.00916: 1.1002x larger Timeline: b'http://tinyurl.com/79koy5q' ### threaded_count ### Min: 0.492332 -> 0.540116: 1.10x slower Avg: 0.518163 -> 0.571137: 1.10x slower Significant (t=-20.88) Stddev: 0.01135 -> 0.01390: 1.2242x larger Timeline: b'http://tinyurl.com/6lsbcvf' ### unpack_sequence ### Min: 0.000208 -> 0.000212: 1.02x slower Avg: 0.000230 -> 0.000243: 1.06x slower Significant (t=-7.71) Stddev: 0.00026 -> 0.00026: 1.0039x larger Timeline: b'http://tinyurl.com/74oeotm' ### mako ### Min: 0.874489 -> 0.823223: 1.06x faster Avg: 0.910595 -> 0.867006: 1.05x faster Significant (t=39.33) Stddev: 0.01187 -> 0.01289: 1.0862x larger Timeline: b'http://tinyurl.com/7zrms7f' ### 2to3 ### 35.266204 -> 35.746234: 1.01x slower ### gc ### 81.385087 -> 72.084505: 1.13x faster The following not significant results are hidden, use -v to show them: call_method, call_method_slots, call_method_unknown, float, nbody, normal_startup, nqueens, pidigits, simple_logging, startup_nosite. Memory ------ ### call_method ### Mem max: 4160.000 -> 4148.000: 1.0029x smaller Usage over time: b'http://tinyurl.com/7vefebw' ### call_method_slots ### Mem max: 4160.000 -> 4144.000: 1.0039x smaller Usage over time: b'http://tinyurl.com/792cyef' ### call_method_unknown ### Mem max: 4460.000 -> 4192.000: 1.0639x smaller Usage over time: b'http://tinyurl.com/6qy28zv' ### call_simple ### Mem max: 4152.000 -> 4124.000: 1.0068x smaller Usage over time: b'http://tinyurl.com/85fjkcd' ### fastpickle ### Mem max: 5112.000 -> 5208.000: 1.0188x larger Usage over time: b'http://tinyurl.com/79fbjo7' ### fastunpickle ### Mem max: 5172.000 -> 5216.000: 1.0085x larger Usage over time: b'http://tinyurl.com/78ux4wb' ### float ### Mem max: 8596.000 -> 6892.000: 1.2472x smaller Usage over time: b'http://tinyurl.com/7m97d7r' ### formatted_logging ### Mem max: 11228.000 -> 11504.000: 1.0246x larger Usage over time: b'http://tinyurl.com/8y8mb64' ### iterative_count ### Mem max: 4208.000 -> 4228.000: 1.0048x larger Usage over time: b'http://tinyurl.com/8834pgs' ### json_dump ### Mem max: 4972.000 -> 4956.000: 1.0032x smaller Usage over time: b'http://tinyurl.com/7y5jlcr' ### json_load ### Mem max: 4984.000 -> 4952.000: 1.0065x smaller Usage over time: b'http://tinyurl.com/87mashx' ### nbody ### Mem max: 4172.000 -> 4160.000: 1.0029x smaller Usage over time: b'http://tinyurl.com/6nlpgr6' ### normal_startup ### Mem max: 3772.000 -> 3744.000: 1.0075x smaller Usage over time: b'http://tinyurl.com/76lbp3j' ### nqueens ### Mem max: 4284.000 -> 4256.000: 1.0066x smaller Usage over time: b'http://tinyurl.com/6lmo2wh' ### pidigits ### Mem max: 4252.000 -> 4236.000: 1.0038x smaller Usage over time: b'http://tinyurl.com/75u5x94' ### regex_compile ### Mem max: 9188.000 -> 8924.000: 1.0296x smaller Usage over time: b'http://tinyurl.com/7zegqhq' ### regex_effbot ### Mem max: 4888.000 -> 4856.000: 1.0066x smaller Usage over time: b'http://tinyurl.com/6trs6gc' ### regex_v8 ### Mem max: 7700.000 -> 8008.000: 1.0400x larger Usage over time: b'http://tinyurl.com/82eole9' ### richards ### Mem max: 4464.000 -> 4492.000: 1.0063x larger Usage over time: b'http://tinyurl.com/6vnhcbo' ### silent_logging ### Mem max: 4844.000 -> 4804.000: 1.0083x smaller Usage over time: b'http://tinyurl.com/7k6l8ty' ### simple_logging ### Mem max: 6860.000 -> 6828.000: 1.0047x smaller Usage over time: b'http://tinyurl.com/7ybn594' ### startup_nosite ### Mem max: 2872.000 -> 2856.000: 1.0056x smaller Usage over time: b'http://tinyurl.com/7z7h3pf' ### threaded_count ### Mem max: 4228.000 -> 4236.000: 1.0019x larger Usage over time: b'http://tinyurl.com/6tdq2mq' ### unpack_sequence ### Mem max: 5704.000 -> 5856.000: 1.0266x larger Usage over time: b'http://tinyurl.com/75sboe3' ### mako ### Mem max: 8796.000 -> 8760.000: 1.0041x smaller Usage over time: b'http://tinyurl.com/6qyt8wm' ### 2to3 ### Mem max: 32676.000 -> 29472.000: 1.1087x smaller Usage over time: b'http://tinyurl.com/7ze4nrv' ### gc ### Mem max: 92460.000 -> 49144.000: 1.8814x smaller Usage over time: b'http://tinyurl.com/7wa9zwg'

Any opinions on my new dictionary implementation? I'm happy to take silence on the PEP as tacit approval, but the code definitely needs reviewing. Issue: http://bugs.python.org/issue13903 PEP: https://bitbucket.org/markshannon/cpython_new_dict/src/6c4d5d9dfc6d/pep-new-... Repository https://bitbucket.org/markshannon/cpython_new_dict Cheers, Mark.

Hello Mark, First, I've back-ported your patch on python 3.2.2 (which was relatively easy). Almost all tests pass, and those that don't are always failing on my machine if I remember. The patch can be found here: http://goo.gl/nSzzY Then, I compared memory footprint of one of our applications (300,000 LOC) and saw it about 6% less than on vanilla python 3.2.2 (660 MB of reserved process memory compared to 702 MB; Linux Gentoo 64bit) The application is written in heavy OOP style (for instance, ~1000 classes are generated by our ORM on the fly, and there are approximately the same amount of hand-written ones) so I hoped for a much bigger saving. As for the patch itself I found one use-case, where python with the patch behaves differently:: class Foo: def __init__(self, msg): self.msg = msg f = Foo('123') class _str(str): pass print(f.msg) print(getattr(f, _str('msg'))) The above snippet works perfectly on vanilla py3.2, but fails on the patched one (even on 3.3 compiled from your 'cpython_new_dict' branch) I'm not sure that it's a valid code, though. If not, then we need to fix some python internals to add exact type check in 'getattr', in the 'operator.getattr', etc. And if it is - your patch needs to be fixed. In any case, I propose to add the above code to the python test-suite, with either expecting a result or an exception. Cheers, Yury On 2012-02-15, at 12:58 PM, Mark Shannon wrote:

On Mon, 13 Feb 2012 12:31:38 +0000 Mark Shannon <mark@hotpy.org> wrote:
Note that the json benchmark is unstable and should be ignored.
Can you elaborate? If it's unstable it should be fixed, not ignored :) Also, there are two different mako results in your message, which one is the right one? Thanks Antoine.

The "type's attribute set" will be a superset of the instance's, for a shared key set. Initializing the first instance grows the key set, which is put into the type. Subsequent instances start out with the key set as a candidate, and have all values set to NULL in the dict values set. As long as you are only setting attributes that are in the shared key set, the values just get set. When it encounters a key not in the shared key set, the dict dissociates itself from the shared key set.
It depends. IIUC, if the first instance happens to get this attribute set, it ends up in the shared key set, and subsequent instances may have a NULL value for the key. I'm unsure how *exactly* the key set gets frozen. You cannot allow resizing the key set once it is shared, as you would have to find all instances with the same key set and resize their values. It would be possible (IIUC) to add more keys to the shared key set if that doesn't cause a resize, but I'm not sure whether the patch does that. Regards, Martin

On 01/02/2012 17:50, Guido van Rossum wrote:
A less common pattern, but which still needs to work, is where a mutable class variable is deliberately store state across all instances of a class... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On 02/02/2012 11:30, Chris Withers wrote:
Given that Mark's patch passes the Python test suite I'm sure basic patterns like this *work*, the question is which of them take advantage of the improved memory efficiency. In the case you mention I don't think it's an issue at all, because the class level attribute doesn't (generally) appear in instance dicts. What's also common is where the class holds a *default* value for instances, which may be overridden by an instance attribute on *some* instances. All the best, Michael Foord
Chris
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Am 02.02.2012 12:30, schrieb Chris Withers:
This is really *just* a dictionary implementation. It doesn't affect any of the lookup procedures. If you trust that the dictionary semantics on its own isn't changed (which I believe is the case, except for key order), none of the dict applications will change. Regards, Martin

On Wed, 1 Feb 2012 09:50:55 -0800 Guido van Rossum <guido@python.org> wrote:
I'm not sure who "you" is in your e-mail, but AFAICT Mark's patch doesn't special-case __init__ or __new__. Any attribute setting on an instance uses the shared keys array on the instance's type. "Missing" attributes on an instance are simply NULL pointers in the instance's values array. (I've suggested that the keys array be bounded in size, to avoid pathological cases where someone (ab)uses instances as fancy dicts and puts lots of random data in them) Regards Antoine.

On 01/29/2012 11:31 AM, Mark Shannon wrote:
It passes all the tests. (I had to change a couple that relied on dict repr() ordering)
Hi Mark, I've cloned the repo, build it the I've tried with ./python -m test. I got some errors: First in general: 340 tests OK. 2 tests failed: test_dis test_gdb 4 tests altered the execution environment: test_multiprocessing test_packaging test_site test_strlit 18 tests skipped: test_curses test_devpoll test_kqueue test_lzma test_msilib test_ossaudiodev test_smtpnet test_socketserver test_startfile test_timeout test_tk test_ttk_guionly test_urllib2net test_urllibnet test_winreg test_winsound test_xmlrpc_net test_zipfile64 1 skip unexpected on linux: test_lzma [1348560 refs] **************************************************** then test_dis: == CPython 3.3.0a0 (default:f15cf35c9922, Jan 29 2012, 18:12:19) [GCC 4.6.2] == Linux-3.1.0-1-amd64-x86_64-with-debian-wheezy-sid little-endian == /home/ci/prog/cpython/hotpy_new_dict/build/test_python_14470 Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0) [1/1] test_dis test_big_linenos (test.test_dis.DisTests) ... ok test_boundaries (test.test_dis.DisTests) ... ok test_bug_1333982 (test.test_dis.DisTests) ... ok test_bug_708901 (test.test_dis.DisTests) ... ok test_dis (test.test_dis.DisTests) ... ok test_dis_none (test.test_dis.DisTests) ... ok test_dis_object (test.test_dis.DisTests) ... ok test_dis_traceback (test.test_dis.DisTests) ... ok test_disassemble_bytes (test.test_dis.DisTests) ... ok test_disassemble_method (test.test_dis.DisTests) ... ok test_disassemble_method_bytes (test.test_dis.DisTests) ... ok test_disassemble_str (test.test_dis.DisTests) ... ok test_opmap (test.test_dis.DisTests) ... ok test_opname (test.test_dis.DisTests) ... ok test_code_info (test.test_dis.CodeInfoTests) ... FAIL test_code_info_object (test.test_dis.CodeInfoTests) ... ok test_pretty_flags_no_flags (test.test_dis.CodeInfoTests) ... ok test_show_code (test.test_dis.CodeInfoTests) ... FAIL ====================================================================== FAIL: test_code_info (test.test_dis.CodeInfoTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/ci/prog/cpython/hotpy_new_dict/Lib/test/test_dis.py", line 439, in test_code_info self.assertRegex(dis.code_info(x), expected) AssertionError: Regex didn't match: 'Name: f\nFilename: (.*)\nArgument count: 1\nKw-only arguments: 0\nNumber of locals: 1\nStack size: 8\nFlags: OPTIMIZED, NEWLOCALS, NESTED\nConstants:\n 0: None\nNames:\n 0: print\nVariable names:\n 0: c\nFree variables:\n 0: e\n 1: d\n 2: f\n 3: y\n 4: x\n 5: z' not found in 'Name: f\nFilename: /home/ci/prog/cpython/hotpy_new_dict/Lib/test/test_dis.py\nArgument count: 1\nKw-only arguments: 0\nNumber of locals: 1\nStack size: 8\nFlags: OPTIMIZED, NEWLOCALS, NESTED\nConstants:\n 0: None\nNames:\n 0: print\nVariable names:\n 0: c\nFree variables:\n 0: y\n 1: e\n 2: d\n 3: f\n 4: x\n 5: z' ====================================================================== FAIL: test_show_code (test.test_dis.CodeInfoTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/ci/prog/cpython/hotpy_new_dict/Lib/test/test_dis.py", line 446, in test_show_code self.assertRegex(output.getvalue(), expected+"\n") AssertionError: Regex didn't match: 'Name: f\nFilename: (.*)\nArgument count: 1\nKw-only arguments: 0\nNumber of locals: 1\nStack size: 8\nFlags: OPTIMIZED, NEWLOCALS, NESTED\nConstants:\n 0: None\nNames:\n 0: print\nVariable names:\n 0: c\nFree variables:\n 0: e\n 1: d\n 2: f\n 3: y\n 4: x\n 5: z\n' not found in 'Name: f\nFilename: /home/ci/prog/cpython/hotpy_new_dict/Lib/test/test_dis.py\nArgument count: 1\nKw-only arguments: 0\nNumber of locals: 1\nStack size: 8\nFlags: OPTIMIZED, NEWLOCALS, NESTED\nConstants:\n 0: None\nNames:\n 0: print\nVariable names:\n 0: c\nFree variables:\n 0: y\n 1: e\n 2: d\n 3: f\n 4: x\n 5: z\n' ---------------------------------------------------------------------- Ran 18 tests in 0.070s FAILED (failures=2) test test_dis failed 1 test failed: test_dis [111919 refs] ***************************************************** For test gdb: Lots of output ..... Ran 42 tests in 11.361s FAILED (failures=28) test test_gdb failed 1 test failed: test_gdb [109989 refs]

francis wrote:
[snip]
**************************************************** then test_dis:
[snip]
These are known failures, the tests are at fault as they rely on dict ordering. However, they should be commented out. Probably crept back in again when I pulled the latest version of cpython -- I'll fix them now. [snip]
I still have gdb 6.somthing, would you mail me the full output please, so I can see what the problem is. Cheers, Mark.

Please clarify the status of that code: are you actually proposing 6a21f3b35e20 for inclusion into Python as-is? If so, please post it as a patch to the tracker, as it will need to be reviewed (possibly with requests for further changes). If not, it would be good if you could give a list of things that need to be done before you consider submission to Python. Also, please submit a contrib form if you haven't done so. Regards, Martin

Martin v. Löwis wrote:
I thought it already was a patch. What do I need to do to make it a patch?
If not, it would be good if you could give a list of things that need to be done before you consider submission to Python.
A few tests that rely on dict ordering should probably be fixed first. I'll submit bug reports for those.
Also, please submit a contrib form if you haven't done so.
Where do I find it? Cheers, Mark.

I missed your announcement of issue13903; all is fine here.
Where do I find it?
http://www.python.org/psf/contrib/contrib-form-python/ Thanks, Martin

Hi, On Sun, 29 Jan 2012 10:31:48 +0000 Mark Shannon <mark@hotpy.org> wrote:
I briefly took a look at your code yesterday and it looked generally reasonable to me. It would be nice to open an issue on http://bugs.python.org so that we can review it there (just fill the "repository" field and use the "create patch" button). Regards Antoine.

2012/1/29 Mark Shannon <mark@hotpy.org>:
If you're serious about changing the dictionary implementation, I think you should write a PEP. It should explain the new dicts advantages (and disadvantages?) and give comprehensive benchmark numbers. Something along the lines of http://www.python.org/dev/peps/pep-3128/ I should think. -- Regards, Benjamin

On Sun, 29 Jan 2012 09:56:11 -0500 Benjamin Peterson <benjamin@python.org> wrote:
"New dictionary implementation" is a misnomer here. Mark's patch merely allows to share the keys array between several dictionaries. The lookup algorithm remains exactly the same as far as I've read. It's actually much less invasive than e.g. Martin's AVL trees-for-hash-collisions proposal. Regards Antoine.

Mark Shannon wrote:
I don't quite follow how that could work. If I have this: class C: pass a = C() b = C() a.spam = 1 b.ham = 2 how can a.__dict__ and b.__dict__ share key arrays? I've tried reading the source, but I'm afraid I don't understand it well enough to make sense of it. -- Steven

On 30/01/12 00:30:14, Steven D'Aprano wrote:
They can't. But then, your class is atypical. Usually, classes initialize all the attributes of their instances in the __init__ method, perhaps like so: class D: def __init__(self, ham=None, spam=None): self.ham = ham self.spam = spam As long as you follow the common practice of not adding any attributes after the object has been initialized, your instances can share their keys array. Mark's patch will do that. You'll still be allowed to have different attributes per instance, but if you do that, then the patch doesn't buy you much. -- HansM

On Wed, Feb 1, 2012 at 9:13 AM, Hans Mulder <hansmu@xs4all.nl> wrote:
Hey, I like this! It's a subtle encouragement for developers to initialize all their instance variables in their __init__ or __new__ method, with a (modest) performance improvement for a carrot. (Though I have to admit I have no idea how you do it. Wouldn't the set of dict keys be different while __init__ is in the middle of setting the instance variables?) Another question: a common pattern is to use (immutable) class variables as default values for instance variables, and only set the instance variables once they need to be different. Does such a class benefit from your improvement?
-- HansM
-- --Guido van Rossum (python.org/~guido)

Guido van Rossum <guido <at> python.org> writes:
While I absolutely cannot speak to this implementation. Traditionally this type of approach is refered to as maps, and was pioneered in SELF, originally presented at OOPSLA '89: http://dl.acm.org/citation.cfm?id=74884 . PyPy also uses these maps to back it's object, although from what I've read the implementation looks nothing like the proposed one for CPython, you can read about that here: http://bit.ly/zwlOkV , and if you're really excited about this you can read our implementation here: https://bitbucket.org/pypy/pypy/src/default/pypy/objspace/std/mapdict.py . Alex

Alex <alex.gaynor <at> gmail.com> writes:
about that here: http://bit.ly/zwlOkV , and if you're really excited about this
Err, here's a working version of the bit.ly link: http://bit.ly/a05h6r Alex

Just a quick update. I've been analysing and profile the behaviour of my new dict and messing about with various implementation options. I've settled on a new implementation. Its the same basic idea, but with better locality of reference for unshared keys. Guido asked:
For those instances which keep the default, yes. Otherwise the answer is, as Martin pointed out, it could yes provided that adding a new key does not force a resize. Although it is a bit arbitrary when a resize occurs. The new version will incorporate this behaviour. Expect version 2 soon. Cheers, Mark.

Hi, Version 2 is now available. Version 2 makes as few changes to tunable constants as possible, and generally does not change iteration order (so repr() is unchanged). All tests pass (the only changes to tests are for sys.getsizeof() ). Repository: https://bitbucket.org/markshannon/cpython_new_dict Issue http://bugs.python.org/issue13903 Performance changes are basically zero for non-OO code. Average -0.5% speed change on 2n3 benchamrks, a few benchmarks show a small reduction in memory use. (see notes below) GCbench uses 47% less memory and is 12% faster. 2to3, which seems to be the only "realistic" benchmark that runs on Py3, shows no change in speed and uses 10% less memory. All benchmarks and tests performed on old, slow 32bit machine with linux. Do please try it on your machine(s). If accepted, the new dict implementation will allow a useful optimisation of the LOAD_GLOBAL (and possibly LOAD_ATTR) bytecode: By testing to see if the (immutable) keys-tables is the expected table, the value can accessed directly by index, rather than by name. Cheers, Mark. Notes: All benchmarks from http://hg.python.org/benchmarks/ using the -m flag to get memory usage data. I've ignored the json benchmarks which shows unstable behaviour on my machine. Tiny changes to the dict being serialized or to the random seed can change the relative speed of my implementation vs CPython from -25% to +10%.

Hi Mark, I've just cloned :
Repository: https://bitbucket.org/markshannon/cpython_new_dict
Do please try it on your machine(s).
.... that's a: Linux random 3.1.0-1-amd64 #1 SMP Tue Jan 10 05:01:58 UTC 2012 x86_64 GNU/Linux and I'm getting: gcc -pthread -c -Wno-unused-result -g -O0 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o Objects/dictobject.o Objects/dictobject.c gcc -pthread -c -Wno-unused-result -g -O0 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o Objects/memoryobject.o Objects/memoryobject.c Objects/dictobject.c: In function ‘dict_popitem’: Objects/dictobject.c:2208:5: error: ‘PyDictKeyEntry’ has no member named ‘me_value’ make: *** [Objects/dictobject.o] Error 1 make: *** Waiting for unfinished jobs.... Cheers francis

Hi Mark,
I've tried to run the test suite but I'm getting a SyntaxError: (may be you know it's just the first time that I try the tool): ============================= ci@random:~/prog/cpython/benchmarks$ python perf.py -r -b apps python ../cpython_new_dict/python Running 2to3... INFO:root:Running ../cpython_new_dict/python lib/2to3/2to3 -f all lib/2to3_data Traceback (most recent call last): File "perf.py", line 2236, in <module> main(sys.argv[1:]) File "perf.py", line 2192, in main options))) File "perf.py", line 1279, in BM_2to3 return SimpleBenchmark(Measure2to3, *args, **kwargs) File "perf.py", line 706, in SimpleBenchmark *args, **kwargs) File "perf.py", line 1275, in Measure2to3 return MeasureCommand(command, trials, env, options.track_memory) File "perf.py", line 1223, in MeasureCommand CallAndCaptureOutput(command, env=env) File "perf.py", line 1053, in CallAndCaptureOutput raise RuntimeError(u"Benchmark died: " + unicode(stderr, 'ascii')) RuntimeError: Benchmark died: Traceback (most recent call last): File "lib/2to3/2to3", line 3, in <module> from lib2to3.main import main File "/home/ci/prog/cpython/benchmarks/lib/2to3/lib2to3/main.py", line 47 except os.error, err: ^ SyntaxError: invalid syntax ============================= And the baseline is: Python 2.7.2+ (but it also gives me an SyntaxError running on python3 default (e50db1b7ad7b) What I'm doing wrong ? (from it's doc: “This project is intended to be an authoritative source of benchmarks for all Python implementations.”) Thanks in advance ! francis

On 08/02/2012 15:16, Mark Shannon wrote:
In your first version 2to3 used 28% less memory. Do you know why it's worse in this version? Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Hi All, My new dictionary implementation has been further updated in light of various comments and questions. The code is here: https://bitbucket.org/markshannon/cpython_new_dict The resizing code has been streamlined. This means that resizing by doubling does not suffer any real performance penalty versus quadrupling. Doubling uses less memory for most benchmarks (it never uses more memory). Michael Foord wrote:
The answer is that the second version used quadrupling rather than doubling for resizing. All tests pass. test_sys has been altered to allow for the different size of the dictionary. One test in test_pprint has been disabled. This test is broken anyway, see http://bugs.python.org/issue13907. In general, for the new dictionary implementation, with doubling at a resize, speed is unchanged and memory usage is reduced. On "average": ~1% slow down, ~10% reduction in memory use. Full set of benchmarks for new dict with doubling and quadrupling attached. Unfortunately the benchmarking program introduces systematic errors for timings, but it's the best we have at the moment. Note that the json benchmark is unstable and should be ignored. The GC benchmark might be unstable as well, I haven't experimented. The memory usage numbers seems to be more reliable. Revised PEP to follow. Cheers, Mark. Report on Linux vespasian 2.6.24-30-generic #1 SMP i686 Total CPU cores: 1 New dict (doubling) vs. CPython =============================== Speed ----- ### call_simple ### Min: 1.131625 -> 1.216112: 1.07x slower Avg: 1.143615 -> 1.223153: 1.07x slower Significant (t=-95.62) Stddev: 0.00694 -> 0.00745: 1.0734x larger Timeline: b'http://tinyurl.com/7ey5unc' ### fastpickle ### Min: 2.365578 -> 2.315522: 1.02x faster Avg: 2.379824 -> 2.326625: 1.02x faster Significant (t=28.84) Stddev: 0.00984 -> 0.00857: 1.1479x smaller Timeline: b'http://tinyurl.com/75nk6fg' ### fastunpickle ### Min: 2.043093 -> 2.083532: 1.02x slower Avg: 2.056331 -> 2.103432: 1.02x slower Significant (t=-31.72) Stddev: 0.00667 -> 0.00811: 1.2144x larger Timeline: b'http://tinyurl.com/7r24oyr' ### float ### Min: 0.256634 -> 0.242119: 1.06x faster Avg: 0.281768 -> 0.252510: 1.12x faster Significant (t=34.46) Stddev: 0.00985 -> 0.00911: 1.0811x smaller Timeline: b'http://tinyurl.com/72cy3ap' ### formatted_logging ### Min: 1.801701 -> 1.886665: 1.05x slower Avg: 1.825188 -> 1.907127: 1.04x slower Significant (t=-31.16) Stddev: 0.01431 -> 0.01188: 1.2046x smaller Timeline: b'http://tinyurl.com/7yjstcs' ### iterative_count ### Min: 0.531924 -> 0.570944: 1.07x slower Avg: 0.549278 -> 0.586633: 1.07x slower Significant (t=-17.09) Stddev: 0.01287 -> 0.00856: 1.5042x smaller Timeline: b'http://tinyurl.com/7afe77x' ### json_dump ### Min: 1.807933 -> 2.361769: 1.31x slower Avg: 1.833581 -> 2.375774: 1.30x slower Significant (t=-249.39) Stddev: 0.01165 -> 0.01003: 1.1623x smaller Timeline: b'http://tinyurl.com/7nkvfdf' ### json_load ### Min: 1.494502 -> 1.733510: 1.16x slower Avg: 1.513454 -> 1.780639: 1.18x slower Significant (t=-50.91) Stddev: 0.01285 -> 0.03481: 2.7088x larger Timeline: b'http://tinyurl.com/78uex2v' ### nbody ### Min: 1.195176 -> 1.111036: 1.08x faster Avg: 1.205988 -> 1.117919: 1.08x faster Significant (t=64.16) Stddev: 0.00770 -> 0.00591: 1.3028x smaller Timeline: b'http://tinyurl.com/88ylc46' ### nqueens ### Min: 1.205363 -> 1.270263: 1.05x slower Avg: 1.219677 -> 1.286260: 1.05x slower Significant (t=-34.55) Stddev: 0.00992 -> 0.00934: 1.0626x smaller Timeline: b'http://tinyurl.com/6nbjq6y' ### regex_compile ### Min: 2.139500 -> 2.410691: 1.13x slower Avg: 2.150651 -> 2.429258: 1.13x slower Significant (t=-168.69) Stddev: 0.00736 -> 0.00907: 1.2333x larger Timeline: b'http://tinyurl.com/88xr386' ### regex_effbot ### Min: 0.194415 -> 0.200979: 1.03x slower Avg: 0.199182 -> 0.206103: 1.03x slower Significant (t=-4.83) Stddev: 0.00715 -> 0.00717: 1.0028x larger Timeline: b'http://tinyurl.com/6ll7pdv' ### regex_v8 ### Min: 0.233819 -> 0.219033: 1.07x faster Avg: 0.249771 -> 0.227338: 1.10x faster Significant (t=7.69) Stddev: 0.01553 -> 0.01358: 1.1440x smaller Timeline: b'http://tinyurl.com/74dqdsy' ### richards ### Min: 0.698549 -> 0.727000: 1.04x slower Avg: 0.717596 -> 0.750720: 1.05x slower Significant (t=-16.90) Stddev: 0.00896 -> 0.01058: 1.1812x larger Timeline: b'http://tinyurl.com/76cvgyj' ### silent_logging ### Min: 0.282622 -> 0.302598: 1.07x slower Avg: 0.291308 -> 0.313256: 1.08x slower Significant (t=-13.24) Stddev: 0.00791 -> 0.00865: 1.0937x larger Timeline: b'http://tinyurl.com/6vovnav' ### simple_logging ### Min: 1.622786 -> 1.796359: 1.11x slower Avg: 1.660321 -> 1.821784: 1.10x slower Significant (t=-38.86) Stddev: 0.02772 -> 0.00974: 2.8450x smaller Timeline: b'http://tinyurl.com/777y27t' ### threaded_count ### Min: 0.501266 -> 0.532309: 1.06x slower Avg: 0.521384 -> 0.549346: 1.05x slower Significant (t=-13.74) Stddev: 0.01074 -> 0.00957: 1.1228x smaller Timeline: b'http://tinyurl.com/889qegb' ### unpack_sequence ### Min: 0.000212 -> 0.000212: no change Avg: 0.000232 -> 0.000242: 1.04x slower Significant (t=-6.07) Stddev: 0.00026 -> 0.00026: 1.0096x larger Timeline: b'http://tinyurl.com/8yzsnmo' ### 2to3 ### 36.702294 -> 37.234327: 1.01x slower ### mako ### Min: 0.982504 -> 0.887246: 1.11x faster Avg: 1.008269 -> 0.934528: 1.08x faster Significant (t=57.62) Stddev: 0.01178 -> 0.01645: 1.3970x larger Timeline: b'http://tinyurl.com/7p2j9s2' ### mako ### Min: 0.753596 -> 0.874137: 1.16x slower Avg: 0.793711 -> 0.915435: 1.15x slower Significant (t=-85.92) Stddev: 0.01607 -> 0.01561: 1.0293x smaller Timeline: b'http://tinyurl.com/8xu5nh9' The following not significant results are hidden, use -v to show them: call_method, call_method_slots, call_method_unknown, normal_startup, pidigits, startup_nosite. Memory ------ ### call_method ### Mem max: 4164.000 -> 4032.000: 1.0327x smaller Usage over time: b'http://tinyurl.com/7saraes' ### call_method_slots ### Mem max: 4164.000 -> 4028.000: 1.0338x smaller Usage over time: b'http://tinyurl.com/722jr7s' ### call_method_unknown ### Mem max: 4464.000 -> 4068.000: 1.0973x smaller Usage over time: b'http://tinyurl.com/6oayz7q' ### call_simple ### Mem max: 4148.000 -> 4008.000: 1.0349x smaller Usage over time: b'http://tinyurl.com/75v3szx' ### fastpickle ### Mem max: 5108.000 -> 4876.000: 1.0476x smaller Usage over time: b'http://tinyurl.com/6s3q7pt' ### fastunpickle ### Mem max: 5132.000 -> 4896.000: 1.0482x smaller Usage over time: b'http://tinyurl.com/7fjhljq' ### float ### Mem max: 8592.000 -> 6772.000: 1.2688x smaller Usage over time: b'http://tinyurl.com/7ke4acp' ### formatted_logging ### Mem max: 10732.000 -> 10892.000: 1.0149x larger Usage over time: b'http://tinyurl.com/83z4f2a' ### iterative_count ### Mem max: 4200.000 -> 4068.000: 1.0324x smaller Usage over time: b'http://tinyurl.com/77ph38p' ### json_dump ### Mem max: 4944.000 -> 4600.000: 1.0748x smaller Usage over time: b'http://tinyurl.com/896y5he' ### json_load ### Mem max: 4940.000 -> 4596.000: 1.0748x smaller Usage over time: b'http://tinyurl.com/77mw6wz' ### nbody ### Mem max: 4172.000 -> 4024.000: 1.0368x smaller Usage over time: b'http://tinyurl.com/7h2795d' ### normal_startup ### Mem max: 3772.000 -> 3632.000: 1.0385x smaller Usage over time: b'http://tinyurl.com/84nwxz7' ### nqueens ### Mem max: 4272.000 -> 4132.000: 1.0339x smaller Usage over time: b'http://tinyurl.com/7sshb2v' ### pidigits ### Mem max: 4248.000 -> 4108.000: 1.0341x smaller Usage over time: b'http://tinyurl.com/82urnw2' ### regex_compile ### Mem max: 8572.000 -> 8904.000: 1.0387x larger Usage over time: b'http://tinyurl.com/6nukmpu' ### regex_effbot ### Mem max: 4864.000 -> 4736.000: 1.0270x smaller Usage over time: b'http://tinyurl.com/767ge37' ### regex_v8 ### Mem max: 7508.000 -> 7860.000: 1.0469x larger Usage over time: b'http://tinyurl.com/72lxz75' ### richards ### Mem max: 4464.000 -> 4356.000: 1.0248x smaller Usage over time: b'http://tinyurl.com/7794rv7' ### silent_logging ### Mem max: 4844.000 -> 4496.000: 1.0774x smaller Usage over time: b'http://tinyurl.com/8332w2a' ### simple_logging ### Mem max: 6952.000 -> 5352.000: 1.2990x smaller Usage over time: b'http://tinyurl.com/8yk7svt' ### startup_nosite ### Mem max: 2872.000 -> 2748.000: 1.0451x smaller Usage over time: b'http://tinyurl.com/72bbdnp' ### threaded_count ### Mem max: 4232.000 -> 4084.000: 1.0362x smaller Usage over time: b'http://tinyurl.com/8yrshqh' ### unpack_sequence ### Mem max: 5688.000 -> 5416.000: 1.0502x smaller Usage over time: b'http://tinyurl.com/7jy6dpl' ### mako ### Mem max: 9012.000 -> 8580.000: 1.0503x smaller Usage over time: b'http://tinyurl.com/82qzfvo' ### 2to3 ### Mem max: 32684.000 -> 22356.000: 1.4620x smaller Usage over time: b'http://tinyurl.com/77hbok6' ### gc ### Mem max: 92600.000 -> 49000.000: 1.8898x smaller Usage over time: b'http://tinyurl.com/86vswmt' New dict (quadrupling) vs. CPython ================================== Speed ----- ### call_simple ### Min: 1.217095 -> 1.269468: 1.04x slower Avg: 1.228556 -> 1.282715: 1.04x slower Significant (t=-55.48) Stddev: 0.00826 -> 0.00864: 1.0459x larger Timeline: b'http://tinyurl.com/7qtz7hk' ### fastpickle ### Min: 2.421091 -> 2.333122: 1.04x faster Avg: 2.447303 -> 2.348845: 1.04x faster Significant (t=41.67) Stddev: 0.01290 -> 0.01062: 1.2142x smaller Timeline: b'http://tinyurl.com/8634dn3' ### fastunpickle ### Min: 2.069723 -> 2.024058: 1.02x faster Avg: 2.082294 -> 2.039120: 1.02x faster Significant (t=17.92) Stddev: 0.00775 -> 0.01517: 1.9572x larger Timeline: b'http://tinyurl.com/73t6ldd' ### formatted_logging ### Min: 1.839560 -> 1.950483: 1.06x slower Avg: 1.868685 -> 1.973164: 1.06x slower Significant (t=-36.79) Stddev: 0.01822 -> 0.00843: 2.1607x smaller Timeline: b'http://tinyurl.com/6pej9zy' ### iterative_count ### Min: 0.482927 -> 0.564538: 1.17x slower Avg: 0.496796 -> 0.575018: 1.16x slower Significant (t=-41.67) Stddev: 0.00982 -> 0.00893: 1.1002x smaller Timeline: b'http://tinyurl.com/735skdg' ### json_dump ### Min: 1.863940 -> 2.130464: 1.14x slower Avg: 1.905174 -> 2.158846: 1.13x slower Significant (t=-44.83) Stddev: 0.03699 -> 0.01525: 2.4253x smaller Timeline: b'http://tinyurl.com/8yx7oso' ### json_load ### Min: 1.586767 -> 1.761993: 1.11x slower Avg: 1.610569 -> 1.793440: 1.11x slower Significant (t=-80.38) Stddev: 0.01279 -> 0.00976: 1.3097x smaller Timeline: b'http://tinyurl.com/6ny4qr5' ### regex_compile ### Min: 2.163662 -> 2.332177: 1.08x slower Avg: 2.174792 -> 2.349151: 1.08x slower Significant (t=-89.97) Stddev: 0.00824 -> 0.01095: 1.3287x larger Timeline: b'http://tinyurl.com/7htwu63' ### regex_effbot ### Min: 0.203670 -> 0.197273: 1.03x faster Avg: 0.208266 -> 0.202605: 1.03x faster Significant (t=3.74) Stddev: 0.00752 -> 0.00763: 1.0143x larger Timeline: b'http://tinyurl.com/7znmo3r' ### regex_v8 ### Min: 0.226723 -> 0.221915: 1.02x faster Avg: 0.239020 -> 0.230629: 1.04x faster Significant (t=3.22) Stddev: 0.01280 -> 0.01325: 1.0349x larger Timeline: b'http://tinyurl.com/7m99e3s' ### richards ### Min: 0.708626 -> 0.727219: 1.03x slower Avg: 0.729989 -> 0.747773: 1.02x slower Significant (t=-8.71) Stddev: 0.01022 -> 0.01020: 1.0020x smaller Timeline: b'http://tinyurl.com/6ueszbh' ### silent_logging ### Min: 0.272886 -> 0.306478: 1.12x slower Avg: 0.281103 -> 0.314645: 1.12x slower Significant (t=-19.17) Stddev: 0.00832 -> 0.00916: 1.1002x larger Timeline: b'http://tinyurl.com/79koy5q' ### threaded_count ### Min: 0.492332 -> 0.540116: 1.10x slower Avg: 0.518163 -> 0.571137: 1.10x slower Significant (t=-20.88) Stddev: 0.01135 -> 0.01390: 1.2242x larger Timeline: b'http://tinyurl.com/6lsbcvf' ### unpack_sequence ### Min: 0.000208 -> 0.000212: 1.02x slower Avg: 0.000230 -> 0.000243: 1.06x slower Significant (t=-7.71) Stddev: 0.00026 -> 0.00026: 1.0039x larger Timeline: b'http://tinyurl.com/74oeotm' ### mako ### Min: 0.874489 -> 0.823223: 1.06x faster Avg: 0.910595 -> 0.867006: 1.05x faster Significant (t=39.33) Stddev: 0.01187 -> 0.01289: 1.0862x larger Timeline: b'http://tinyurl.com/7zrms7f' ### 2to3 ### 35.266204 -> 35.746234: 1.01x slower ### gc ### 81.385087 -> 72.084505: 1.13x faster The following not significant results are hidden, use -v to show them: call_method, call_method_slots, call_method_unknown, float, nbody, normal_startup, nqueens, pidigits, simple_logging, startup_nosite. Memory ------ ### call_method ### Mem max: 4160.000 -> 4148.000: 1.0029x smaller Usage over time: b'http://tinyurl.com/7vefebw' ### call_method_slots ### Mem max: 4160.000 -> 4144.000: 1.0039x smaller Usage over time: b'http://tinyurl.com/792cyef' ### call_method_unknown ### Mem max: 4460.000 -> 4192.000: 1.0639x smaller Usage over time: b'http://tinyurl.com/6qy28zv' ### call_simple ### Mem max: 4152.000 -> 4124.000: 1.0068x smaller Usage over time: b'http://tinyurl.com/85fjkcd' ### fastpickle ### Mem max: 5112.000 -> 5208.000: 1.0188x larger Usage over time: b'http://tinyurl.com/79fbjo7' ### fastunpickle ### Mem max: 5172.000 -> 5216.000: 1.0085x larger Usage over time: b'http://tinyurl.com/78ux4wb' ### float ### Mem max: 8596.000 -> 6892.000: 1.2472x smaller Usage over time: b'http://tinyurl.com/7m97d7r' ### formatted_logging ### Mem max: 11228.000 -> 11504.000: 1.0246x larger Usage over time: b'http://tinyurl.com/8y8mb64' ### iterative_count ### Mem max: 4208.000 -> 4228.000: 1.0048x larger Usage over time: b'http://tinyurl.com/8834pgs' ### json_dump ### Mem max: 4972.000 -> 4956.000: 1.0032x smaller Usage over time: b'http://tinyurl.com/7y5jlcr' ### json_load ### Mem max: 4984.000 -> 4952.000: 1.0065x smaller Usage over time: b'http://tinyurl.com/87mashx' ### nbody ### Mem max: 4172.000 -> 4160.000: 1.0029x smaller Usage over time: b'http://tinyurl.com/6nlpgr6' ### normal_startup ### Mem max: 3772.000 -> 3744.000: 1.0075x smaller Usage over time: b'http://tinyurl.com/76lbp3j' ### nqueens ### Mem max: 4284.000 -> 4256.000: 1.0066x smaller Usage over time: b'http://tinyurl.com/6lmo2wh' ### pidigits ### Mem max: 4252.000 -> 4236.000: 1.0038x smaller Usage over time: b'http://tinyurl.com/75u5x94' ### regex_compile ### Mem max: 9188.000 -> 8924.000: 1.0296x smaller Usage over time: b'http://tinyurl.com/7zegqhq' ### regex_effbot ### Mem max: 4888.000 -> 4856.000: 1.0066x smaller Usage over time: b'http://tinyurl.com/6trs6gc' ### regex_v8 ### Mem max: 7700.000 -> 8008.000: 1.0400x larger Usage over time: b'http://tinyurl.com/82eole9' ### richards ### Mem max: 4464.000 -> 4492.000: 1.0063x larger Usage over time: b'http://tinyurl.com/6vnhcbo' ### silent_logging ### Mem max: 4844.000 -> 4804.000: 1.0083x smaller Usage over time: b'http://tinyurl.com/7k6l8ty' ### simple_logging ### Mem max: 6860.000 -> 6828.000: 1.0047x smaller Usage over time: b'http://tinyurl.com/7ybn594' ### startup_nosite ### Mem max: 2872.000 -> 2856.000: 1.0056x smaller Usage over time: b'http://tinyurl.com/7z7h3pf' ### threaded_count ### Mem max: 4228.000 -> 4236.000: 1.0019x larger Usage over time: b'http://tinyurl.com/6tdq2mq' ### unpack_sequence ### Mem max: 5704.000 -> 5856.000: 1.0266x larger Usage over time: b'http://tinyurl.com/75sboe3' ### mako ### Mem max: 8796.000 -> 8760.000: 1.0041x smaller Usage over time: b'http://tinyurl.com/6qyt8wm' ### 2to3 ### Mem max: 32676.000 -> 29472.000: 1.1087x smaller Usage over time: b'http://tinyurl.com/7ze4nrv' ### gc ### Mem max: 92460.000 -> 49144.000: 1.8814x smaller Usage over time: b'http://tinyurl.com/7wa9zwg'

Any opinions on my new dictionary implementation? I'm happy to take silence on the PEP as tacit approval, but the code definitely needs reviewing. Issue: http://bugs.python.org/issue13903 PEP: https://bitbucket.org/markshannon/cpython_new_dict/src/6c4d5d9dfc6d/pep-new-... Repository https://bitbucket.org/markshannon/cpython_new_dict Cheers, Mark.

Hello Mark, First, I've back-ported your patch on python 3.2.2 (which was relatively easy). Almost all tests pass, and those that don't are always failing on my machine if I remember. The patch can be found here: http://goo.gl/nSzzY Then, I compared memory footprint of one of our applications (300,000 LOC) and saw it about 6% less than on vanilla python 3.2.2 (660 MB of reserved process memory compared to 702 MB; Linux Gentoo 64bit) The application is written in heavy OOP style (for instance, ~1000 classes are generated by our ORM on the fly, and there are approximately the same amount of hand-written ones) so I hoped for a much bigger saving. As for the patch itself I found one use-case, where python with the patch behaves differently:: class Foo: def __init__(self, msg): self.msg = msg f = Foo('123') class _str(str): pass print(f.msg) print(getattr(f, _str('msg'))) The above snippet works perfectly on vanilla py3.2, but fails on the patched one (even on 3.3 compiled from your 'cpython_new_dict' branch) I'm not sure that it's a valid code, though. If not, then we need to fix some python internals to add exact type check in 'getattr', in the 'operator.getattr', etc. And if it is - your patch needs to be fixed. In any case, I propose to add the above code to the python test-suite, with either expecting a result or an exception. Cheers, Yury On 2012-02-15, at 12:58 PM, Mark Shannon wrote:

On Mon, 13 Feb 2012 12:31:38 +0000 Mark Shannon <mark@hotpy.org> wrote:
Note that the json benchmark is unstable and should be ignored.
Can you elaborate? If it's unstable it should be fixed, not ignored :) Also, there are two different mako results in your message, which one is the right one? Thanks Antoine.

The "type's attribute set" will be a superset of the instance's, for a shared key set. Initializing the first instance grows the key set, which is put into the type. Subsequent instances start out with the key set as a candidate, and have all values set to NULL in the dict values set. As long as you are only setting attributes that are in the shared key set, the values just get set. When it encounters a key not in the shared key set, the dict dissociates itself from the shared key set.
It depends. IIUC, if the first instance happens to get this attribute set, it ends up in the shared key set, and subsequent instances may have a NULL value for the key. I'm unsure how *exactly* the key set gets frozen. You cannot allow resizing the key set once it is shared, as you would have to find all instances with the same key set and resize their values. It would be possible (IIUC) to add more keys to the shared key set if that doesn't cause a resize, but I'm not sure whether the patch does that. Regards, Martin

On 01/02/2012 17:50, Guido van Rossum wrote:
A less common pattern, but which still needs to work, is where a mutable class variable is deliberately store state across all instances of a class... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On 02/02/2012 11:30, Chris Withers wrote:
Given that Mark's patch passes the Python test suite I'm sure basic patterns like this *work*, the question is which of them take advantage of the improved memory efficiency. In the case you mention I don't think it's an issue at all, because the class level attribute doesn't (generally) appear in instance dicts. What's also common is where the class holds a *default* value for instances, which may be overridden by an instance attribute on *some* instances. All the best, Michael Foord
Chris
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Am 02.02.2012 12:30, schrieb Chris Withers:
This is really *just* a dictionary implementation. It doesn't affect any of the lookup procedures. If you trust that the dictionary semantics on its own isn't changed (which I believe is the case, except for key order), none of the dict applications will change. Regards, Martin

On Wed, 1 Feb 2012 09:50:55 -0800 Guido van Rossum <guido@python.org> wrote:
I'm not sure who "you" is in your e-mail, but AFAICT Mark's patch doesn't special-case __init__ or __new__. Any attribute setting on an instance uses the shared keys array on the instance's type. "Missing" attributes on an instance are simply NULL pointers in the instance's values array. (I've suggested that the keys array be bounded in size, to avoid pathological cases where someone (ab)uses instances as fancy dicts and puts lots of random data in them) Regards Antoine.

On 01/29/2012 11:31 AM, Mark Shannon wrote:
It passes all the tests. (I had to change a couple that relied on dict repr() ordering)
Hi Mark, I've cloned the repo, build it the I've tried with ./python -m test. I got some errors: First in general: 340 tests OK. 2 tests failed: test_dis test_gdb 4 tests altered the execution environment: test_multiprocessing test_packaging test_site test_strlit 18 tests skipped: test_curses test_devpoll test_kqueue test_lzma test_msilib test_ossaudiodev test_smtpnet test_socketserver test_startfile test_timeout test_tk test_ttk_guionly test_urllib2net test_urllibnet test_winreg test_winsound test_xmlrpc_net test_zipfile64 1 skip unexpected on linux: test_lzma [1348560 refs] **************************************************** then test_dis: == CPython 3.3.0a0 (default:f15cf35c9922, Jan 29 2012, 18:12:19) [GCC 4.6.2] == Linux-3.1.0-1-amd64-x86_64-with-debian-wheezy-sid little-endian == /home/ci/prog/cpython/hotpy_new_dict/build/test_python_14470 Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0) [1/1] test_dis test_big_linenos (test.test_dis.DisTests) ... ok test_boundaries (test.test_dis.DisTests) ... ok test_bug_1333982 (test.test_dis.DisTests) ... ok test_bug_708901 (test.test_dis.DisTests) ... ok test_dis (test.test_dis.DisTests) ... ok test_dis_none (test.test_dis.DisTests) ... ok test_dis_object (test.test_dis.DisTests) ... ok test_dis_traceback (test.test_dis.DisTests) ... ok test_disassemble_bytes (test.test_dis.DisTests) ... ok test_disassemble_method (test.test_dis.DisTests) ... ok test_disassemble_method_bytes (test.test_dis.DisTests) ... ok test_disassemble_str (test.test_dis.DisTests) ... ok test_opmap (test.test_dis.DisTests) ... ok test_opname (test.test_dis.DisTests) ... ok test_code_info (test.test_dis.CodeInfoTests) ... FAIL test_code_info_object (test.test_dis.CodeInfoTests) ... ok test_pretty_flags_no_flags (test.test_dis.CodeInfoTests) ... ok test_show_code (test.test_dis.CodeInfoTests) ... FAIL ====================================================================== FAIL: test_code_info (test.test_dis.CodeInfoTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/ci/prog/cpython/hotpy_new_dict/Lib/test/test_dis.py", line 439, in test_code_info self.assertRegex(dis.code_info(x), expected) AssertionError: Regex didn't match: 'Name: f\nFilename: (.*)\nArgument count: 1\nKw-only arguments: 0\nNumber of locals: 1\nStack size: 8\nFlags: OPTIMIZED, NEWLOCALS, NESTED\nConstants:\n 0: None\nNames:\n 0: print\nVariable names:\n 0: c\nFree variables:\n 0: e\n 1: d\n 2: f\n 3: y\n 4: x\n 5: z' not found in 'Name: f\nFilename: /home/ci/prog/cpython/hotpy_new_dict/Lib/test/test_dis.py\nArgument count: 1\nKw-only arguments: 0\nNumber of locals: 1\nStack size: 8\nFlags: OPTIMIZED, NEWLOCALS, NESTED\nConstants:\n 0: None\nNames:\n 0: print\nVariable names:\n 0: c\nFree variables:\n 0: y\n 1: e\n 2: d\n 3: f\n 4: x\n 5: z' ====================================================================== FAIL: test_show_code (test.test_dis.CodeInfoTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/ci/prog/cpython/hotpy_new_dict/Lib/test/test_dis.py", line 446, in test_show_code self.assertRegex(output.getvalue(), expected+"\n") AssertionError: Regex didn't match: 'Name: f\nFilename: (.*)\nArgument count: 1\nKw-only arguments: 0\nNumber of locals: 1\nStack size: 8\nFlags: OPTIMIZED, NEWLOCALS, NESTED\nConstants:\n 0: None\nNames:\n 0: print\nVariable names:\n 0: c\nFree variables:\n 0: e\n 1: d\n 2: f\n 3: y\n 4: x\n 5: z\n' not found in 'Name: f\nFilename: /home/ci/prog/cpython/hotpy_new_dict/Lib/test/test_dis.py\nArgument count: 1\nKw-only arguments: 0\nNumber of locals: 1\nStack size: 8\nFlags: OPTIMIZED, NEWLOCALS, NESTED\nConstants:\n 0: None\nNames:\n 0: print\nVariable names:\n 0: c\nFree variables:\n 0: y\n 1: e\n 2: d\n 3: f\n 4: x\n 5: z\n' ---------------------------------------------------------------------- Ran 18 tests in 0.070s FAILED (failures=2) test test_dis failed 1 test failed: test_dis [111919 refs] ***************************************************** For test gdb: Lots of output ..... Ran 42 tests in 11.361s FAILED (failures=28) test test_gdb failed 1 test failed: test_gdb [109989 refs]

francis wrote:
[snip]
**************************************************** then test_dis:
[snip]
These are known failures, the tests are at fault as they rely on dict ordering. However, they should be commented out. Probably crept back in again when I pulled the latest version of cpython -- I'll fix them now. [snip]
I still have gdb 6.somthing, would you mail me the full output please, so I can see what the problem is. Cheers, Mark.

Please clarify the status of that code: are you actually proposing 6a21f3b35e20 for inclusion into Python as-is? If so, please post it as a patch to the tracker, as it will need to be reviewed (possibly with requests for further changes). If not, it would be good if you could give a list of things that need to be done before you consider submission to Python. Also, please submit a contrib form if you haven't done so. Regards, Martin

Martin v. Löwis wrote:
I thought it already was a patch. What do I need to do to make it a patch?
If not, it would be good if you could give a list of things that need to be done before you consider submission to Python.
A few tests that rely on dict ordering should probably be fixed first. I'll submit bug reports for those.
Also, please submit a contrib form if you haven't done so.
Where do I find it? Cheers, Mark.

I missed your announcement of issue13903; all is fine here.
Where do I find it?
http://www.python.org/psf/contrib/contrib-form-python/ Thanks, Martin
participants (14)
-
"Martin v. Löwis"
-
Alex
-
Alex Gaynor
-
Antoine Pitrou
-
Benjamin Peterson
-
Chris Withers
-
francis
-
Guido van Rossum
-
Hans Mulder
-
Mark Shannon
-
martin@v.loewis.de
-
Michael Foord
-
Steven D'Aprano
-
Yury Selivanov