We released ZODB 3.2b2 on Tuesday evening. You can find it at the usual
place: http://www.zope.org/Products/ZODB3.2
This release contains the last new work we plan to do before the ZODB
3.2 final release. We have fixed several bugs, including a deadlock
problem that had been reported against 3.2b2.
ZODB 3.2 has a number of new features and improvements over ZODB 3.1:
- improve performance and stability of ZEO
- new ZEO authentication protocol
- new configuration language, ZConfig, for databases, storages,
and ZEO servers
- many bug fixes
Please give this new release a try. We hope to have a final release
soon, and bugs reported soon are more likely to get fixed before
the final release.
I've included the NEWS.txt summary of the most recent changes.
Jeremy
What's new in ZODB3 3.2 beta 3
==============================
Release date: 23-Sep-2003
Note: The changes listed for this release include changes also made in
ZODB 3.1.x releases and ported to the 3.2 release.
This version of ZODB 3.2 is not compatible with Python 2.1. Early
versions were explicitly designed to be compatible with Zope 2.6.
That plan has been dropped, because Zope 2.7 is already in beta
release.
Several of the classes in ZEO and ZODB now inherit from object, making
them new-style classes. The primary motivation for the change was to
make it easier to debug memory leaks. We don't expect any behavior to
change as a result.
A new feature to allow removal of connection pools for versions was
ported from Zope 2.6. This feature is needed by Zope to avoid denial
of service attacks that allow a client to create an arbitrary number
of version pools.
Fixed several critical ZEO bugs.
- If several client transactions were blocked waiting for the storage
and one of the blocked clients disconnected, the server would
attempt to restart one of the other waiting clients. Since the
disconnected client did not have the storage lock, this could lead
to deadlock. It could also cause the assertion "self._client is
None" to fail.
- If a storage server fails or times out between the vote and the
finish, the ZEO cache could get populated with objects that didn't
make it to the storage server.
- If a client loses its connection to the server near the end of a
transaction, it is now guaranteed to get a ClientDisconnected error
even if it reconnects before the transaction finishes. This is
necessary because the server will always abort the transaction.
In some cases, the client would never see an error for the aborted
transaction.
- In tpc_finish(), reordered the calls so that the server's tpc_finish()
is called (and must succeed) before we update the ZEO client cache.
- The storage name is now prepended to the sort key, to ensure a
unique global sort order if storages are named uniquely. This
can prevent deadlock in some unusual cases.
Fixed several serious flaws in the implementation of the ZEO
authentication protocol.
- The smac layer would accept a message without a MAC even after the
session key was established.
- The client never initialized its session key, so it never checked
incoming messages or created MACs for outgoing messags.
- The smac layer used a single HMAC instance for sending and receiving
messages. This approach could only work if client and server were
guaranteed to process all messages in the same total order, which
could only happen in simple scenarios like unit tests.
Fixed a bug in ExtensionClass when comparing ExtensionClass instances.
The code could raise RuntimeWarning under Python 2.3, and produce
incorrect results on 64-bit platforms.
Fixed bug in BDBStorage that cause lead to DBRunRecoveryErrors when a
transaction was aborted after performing operations like commit
version or undo that create new references to existing pickles.
Fixed a bug in Connection.py that caused it to fail with an
AttributeError if close() was called after the database was closed.
The test suite leaves fewer log files behind, although it still leaves
a lot of junk. The test.py script puts each tests temp files in a
separate directory, so it is easier to see which tests are causing
problems. Unfortunately, it is still to tedious to figure out why the
identified tests are leaving files behind.
This release contains the latest and greatest version of the
BDBStorage. This storage has still not seen testing in a production
environment, but it represents the current best design and most recent
code culled from various branches where development has occurred.
The Tools directory contains a number of small improvements, a few new
tools, and README.txt that catalogs the tools. Many of the tools are
installed by setup.py; those scripts will now have a #! line set
automatically on Unix.
Fixed bugs in Tools/repozo.py, including a timing-dependent one that
could cause the following invocation of repozo to do a full backup when
an incremental backup would have sufficed.
A pair of new scripts from Jim Fulton can be used to synthesize
workloads and measure ZEO performance: see zodbload.py and
zeoserverlog.py in the Tools directory. Note that these require
Zope.
Tools/checkbtrees.py was strengthened in two ways:
- In addition to running the _check() method on each BTree B found,
BTrees.check.check(B) is also run. The check() function was written
after checkbtrees.py, and identifies kinds of damage B._check()
cannot find.
- Cycles in the object graph no longer lead to unbounded output.
Note that preventing this requires remembering the oid of each
persistent object found, which increases the memory needed by the
script.
We at NeuroKode Labs, LLC, are proud to announce the release of Python
Database Objects (PDO) 1.0.1. The current release adds support for
PostgreSQL
via the pgdb Module.
As released previously: Python Database Objects (PDO) provides an easy to
use Object
Oriented API for database developers. PDO utilizes DB-API modules for
database access,
but allows for a Common Object Oriented API across RDBMS. Thus, PDO can be
thought of
as a 'wrapper' around the DB-API and database specific modules.
PDO offers a unique interface which includes such features as column
access by name,
forward and backward movement through a result set and much, much more.
Downloads for Python Database Objects are available on SourceForge.Net
or for more information please visit pdo.neurokode.com.
~Bryan J Gudorf
NeuroKode Labs, LLC
I am holding another 3-day Python class
in Longmont, Colorado, on October 7-9.
This is a public class, open to individual
enrollments, and covers the same topics as
the on-site classes I teach. For more
details, please see this page:
http://www.rmi.net/~lutz/oct03-longmont-class.html
Thanks for your interest.
--Mark Lutz (http://www.rmi.net/~lutz)
Announcing PyTables 0.7.2
-------------------------
PyTables is a hierarchical database package designed to efficently
manage very large amounts of data. PyTables is built on top of the
HDF5 library and the numarray package. It features an object-oriented
interface that, combined with natural naming and C-code generated from
Pyrex sources, makes it a fast, yet extremely easy to use tool for
interactively save and retrieve large amounts of data. Besides, it
provides flexible indexed access on disk to anywhere in the data you
want to go.
On this release you will not find any exciting new features. It is
mainly a maintenance release where the next issues has been addressed:
- a memory leak was fixed
- memory needs is being lowered
- much faster opening of files
- done some important optimizations in table reads
More in detail:
What's new
-----------
- Fixed a nasty memory leak located on the C libraries (it was
happening during HDF5 attribute writes). After that, the
memory consumption when using large object trees has dropped
quite a bit. However, there remains some small leaks that
has been tracked down to the underlying numarray
library. These leaks has been reported, and hopefully they
should be fixed more sooner than later.
- Table buffers are built dinamically now, so if Tables are
not accessed for reading or writing this memory will not be
booked. This will help to reduce the memory consumption.
- The opening of files with lots of nodes has been accelerated
between a factor 2 or 3. For example, a file with 10 groups
and 3000 tables that takes 9.3 seconds to open in 0.7.1, now
takes only 2.8 seconds.
- The Table.read() method has been refactored and optimized
and some parts of its code has been moved to Pyrex. In
particular, in the special case of step=1, up to a factor 5
of speedup (reaching 160 MB/s on a Pentium4 @ 2 GHz) when
reading table contents can be achieved now.
- Done some cosmetic changes in the user manual, but, as no
new features has been added, you won't need to read the
manual again :-)
What is a table?
----------------
A table is defined as a collection of records whose values are stored
in fixed-length fields. All records have the same structure and all
values in each field have the same data type. The terms
"fixed-length" and "strict data types" seems to be quite a strange
requirement for an language like Python, that supports dynamic data
types, but they serve a useful function if the goal is to save very
large quantities of data (such as is generated by many scientific
applications, for example) in an efficient manner that reduces demand
on CPU time and I/O resources.
What is HDF5?
-------------
For those people who know nothing about HDF5, it is is a general
purpose library and file format for storing scientific data made at
NCSA. HDF5 can store two primary objects: datasets and groups. A
dataset is essentially a multidimensional array of data elements, and
a group is a structure for organizing objects in an HDF5 file. Using
these two basic constructs, one can create and store almost any kind of
scientific data structure, such as images, arrays of vectors, and
structured and unstructured grids. You can also mix and match them in
HDF5 files according to your needs.
Platforms
---------
I'm using Linux as the main development platform, but PyTables should
be easy to compile/install on other UNIX machines. This package has
also passed all the tests on a UltraSparc platform with Solaris 7 and
Solaris 8. It also compiles and passes all the tests on a SGI
Origin2000 with MIPS R12000 processors and running IRIX 6.5.
Regarding Windows platforms, PyTables has been tested with Windows
2000 and Windows XP, but it should also work with other flavors.
An example?
-----------
For online code examples, have a look at
http://pytables.sourceforge.net/tut/tutorial1-1.html
and
http://pytables.sourceforge.net/tut/tutorial1-2.html
Web site
--------
Go to the PyTables web site for more details:
http://pytables.sourceforge.net/
Share your experience
---------------------
Let me know of any bugs, suggestions, gripes, kudos, etc. you may
have.
Have fun!
--
Francesc Alted
python-dev Summary for 2003-09-01 through 2003-09-15
++++++++++++++++++++++++++++++++++++++++++++++++++++
This is a summary of traffic on the `python-dev mailing list`_ from
September 1, 2003 through September 15, 2003. It is intended to inform
the wider Python community of on-going developments on the list. To
comment on anything mentioned here, just post to `comp.lang.python`_ (or
email python-list(a)python.org which is a gateway to the newsgroup) with a
subject line mentioning what you are discussing. All python-dev members
are interested in seeing ideas discussed by the community, so don't
hesitate to take a stance on something. And if all of this really
interests you then get involved and join `python-dev`_!
This is the twenty-fifth summary written by Brett Cannon (with school
looming on the horizon).
All summaries are archived at http://www.python.org/dev/summary/ .
Please note that this summary is written using reStructuredText_ which
can be found at http://docutils.sf.net/rst.html . Any unfamiliar
punctuation is probably markup for reST_ (otherwise it is probably
regular expression syntax or a typo =); you can safely ignore it,
although I suggest learning reST; it's simple and is accepted for `PEP
markup`_ and gives some perks for the HTML output. Also, because of the
wonders of programs that like to reformat text, I cannot guarantee you
will be able to run the text version of this summary through Docutils_
as-is unless it is from the original text file.
.. _PEP Markup: http://www.python.org/peps/pep-0012.html
The in-development version of the documentation for Python can be found
at http://www.python.org/dev/doc/devel/ and should be used when looking
up any documentation on something mentioned here. PEPs (Python
Enhancement Proposals) are located at http://www.python.org/peps/ . To
view files in the Python CVS online, go to
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/ . Reported bugs
and suggested patches can be found at the SourceForge_ project page.
.. _python-dev: http://www.python.org/dev/
.. _SourceForge: http://sourceforge.net/tracker/?group_id=5470
.. _python-dev mailing list:
http://mail.python.org/mailman/listinfo/python-dev
.. _comp.lang.python: http://groups.google.com/groups?q=comp.lang.python
.. _Docutils: http://docutils.sf.net/
.. _reST:
.. _reStructuredText: http://docutils.sf.net/rst.html
.. contents::
.. _last summary:
http://www.python.org/dev/summary/2003-08-16_2003-08-31.html
=====================
Summary Announcements
=====================
Summary is nice and short this time. More people went on vacation
(although Martin came back and so that helped to pick up the traffic a
little). I skipped covering some threads that are due for PEPs since
those will be more in-depth then anything I write.
I should give fair warning that starting on the 22nd of September I will
begin school. I am hoping that it will not be difficult and I will have
a carefree time in school. But if my workload becomes a burden the
Summaries might suffer. I hope they don't, though, and I am going to do
my best to make sure that doesn't happen.
=========
Summaries
=========
---------------------------------------
If It Isn't Documented, Is It a Secret?
---------------------------------------
Some undocumented methods in readline were discovered by Philip Eby. He
asked if this was on purpose and whether he should submit a bug report
on the lack of documentation. Guido told him to go ahead and submit the
bug report.
If you ever find something undocumented in the Python source code and
there is nothing suggesting the code is meant to be hidden from the user
(it's for internal use, a comment says it is experimental, etc.), then
please file a bug report!
Contributing threads:
- `Undocumented functions in 'readline' module
<http://mail.python.org/pipermail/python-dev/2003-September/037922.html>`__
-----------------------------------------------------------------------------------------
"You are more trouble than you are worth!", screamed the micro release
to the new feature
-----------------------------------------------------------------------------------------
The topic of what should be backported for micro releases (with Python
2.3.1 looming on the horizon) came up. Originally minor improvements
were allowed in if they didn't break backwards compatibility. And of
course bugfixes have been as well as long as they didn't break code that
came to depend on the buggy behavior (really depends on how long the
bugginess has been there and how well-known it is).
But then the point of OS X 10.3 possibly becoming the largest install
base of Python of any version (it will be 2.3) came up. With rough
estimates being thrown around of 5 million installs in about a year's
time, the point that making it difficult to run Python 2.3.x code on
that size of an install base would be bad. For instance, if some small
new feature was added to 2.3.1 then any code using that feature would
not be able to run on a virgin install of OS X 10.3 (and considering
that this is Mac it should not be expected that most users will want to,
let alone know how, to add a secondary install of Python since the
original is used by the OS and thus should not be overwritten). It
looks like a more conservative patching scheme will be taken with micro
releases.
Bytecode will also continue to work between releases. Guido has always
unofficially held that position but now it has been vocalized.
Contributing threads:
- `Documenting branch policy
<http://mail.python.org/pipermail/python-dev/2003-September/037955.html>`__
----------------------------------------------
PyPy "Berlin" sprint (29th Sep - 4th Oct 2003)
----------------------------------------------
Just what the title says: there is going to be a sprint_ for PyPy_ in
Berlin from September 29 to October 4. Read the email for more specific
details on goals and such.
.. _sprint: http://www.python.org/cgi-bin/moinmoin/SprintPlan
.. _PyPy: http://codespeak.net/pypy/index.cgi?home
Contributing threads:
- `PyPy "Berlin" sprint (29th Sep - 4th Oct 2003)
<http://mail.python.org/pipermail/python-dev/2003-September/037961.html>`__
--------------------------------
Away with you ambiguous imports!
--------------------------------
A discussion on how to create a hierarchy of loggers in the standard
library led to the idea of having a way to cause an import to come
directly from the standard library and thus remove any ambiguity from a
relative import grabbing a similarly named module from the local package.
Barry Warsaw said he was thinking of writing a PEP on this idea.
Contributing threads:
- `Consistent logging in the standard library
<http://mail.python.org/pipermail/python-dev/2003-September/038000.html>`__
---------------------------------------------------
Quick: want to give a Python talk at Linuxworld NY?
---------------------------------------------------
If you are interested in what the title asks, then read the email for
some details.
Contributing threads:
- `Quick: want to give a Python talk at Linuxworld NY?
<http://mail.python.org/pipermail/python-dev/2003-September/037993.html>`__
-------------------------------------------
Be careful about saying something is "easy"
-------------------------------------------
A word of advice about saying something is "easy" on python-dev: if you
say that you should make sure you can back up that claim because
otherwise you will be told to write the "easy" patch and that will be
the end of the discussion.
Contributing threads:
- `Making python C-API thread safe (try 2)
<http://mail.python.org/pipermail/python-dev/2003-September/038007.html>`__
--------------------------
Parsing dates for datetime
--------------------------
There was some talk about coming up with some code to parse dates for
the datetime module. It was kind of hand-wavy since there are multiples
chunks of code that can do that in the standard library.
If you have an opinion on this (or anything for that matter), make sure
to make a post to `comp.lang.python`_ about it.
Contributing threads:
- `datetime issues
<http://mail.python.org/pipermail/python-dev/2003-September/038042.html>`__
http://wwwsearch.sourceforge.net/ClientForm/http://wwwsearch.sourceforge.net/ClientForm/src/README-0_1_8b.html
Second (and last, I hope) beta release of 0.1.x.
Changes from 0.1.7a to 0.1.8b:
* Interface change (sorry): id is now directly supported. Controls have
an id attribute, and appropriate HTMLForm methods have an id argument.
This will only affect code that uses the positional arguments that come
after the 'kind' argument of the various HTMLForm methods.
* Interface change: BUTTON/BUTTON now has type "buttonbutton" (was
"button") to prevent clash with type of INPUT/BUTTON (was and is
"button"). Both types of control are ignored anyway (ie. represented
by IgnoreControl), so it's unlikely any code is affected.
* SubmitButton value now defaults to "", so it is successful even when no
value is given in the HTML.
* File upload bugs fixed.
* Minor bug fixes.
Requires Python >= 1.5.2.
ClientForm is a Python module for handling HTML forms on the client
side, useful for parsing HTML forms, filling them in and returning the
completed forms to the server. It has developed from a port of Gisle
Aas' Perl module HTML::Form, from the libwww-perl library, but the
interface is not the same.
Simple example:
from urllib2 import urlopen
from ClientForm import ParseResponse
forms = ParseResponse(urlopen("http://www.acme.com/form.html"))
form = forms[0]
print form
form["author"] = "Gisle Aas"
# form.click returns a urllib2.Request object
# (see HTMLForm.click_request_data.__doc__ if you're not using urllib2)
response = urlopen(form.click("Thanks"))
John
pyTerra is a Python module that allows you to make requests to Microsoft's
TerraServer. With it, you can download cartographic images for any almost
any geographic extent in the conterminous US. It mimics the SOAP API
provided by the TerraServer located at
<http://terraservice.net/TerraService.asmx>. A helper class is also
provided that reduces the interaction required to defining a bounding box
and writing the image out to the file system.
A source distribution and Windows binary is provided.
See <http://hobu.biz/software/pyTerra> for more information and download.
Howard Butler
mgSeek
-------
imgSeek is a photo collection manager and viewer with content-based
search and many other features. The query is expressed either as a
rough sketch painted by the user or as another image you supply (or an
image in your collection).
You may also do slideshows, generate web photo albums, edit image
metadata including EXIF and IPTC data, organize images into a keyword
hierarchy, and more.
Changes
-------
A bug related to settings prevented users with python2.3 from viewing
images on the preview window, so 0.8.2 sole purpose is to fix that.
Not much development effort has been put lately on new imgSeek
features, so expect only bugfixes to be released in the next few
months.
Meanwhile we've been working on the imgSeek-net sub-project
(http://imgseek.sourceforge.net/net/ -- 'imgseek-net' cvs module) and
on the 'SQL' cvs branch of imgSeek, whose long term objective would be
the usage of popular RDBMS as a transparent replacement for the
current implementation of a grouping and metadata database on top of
python's 'marshal' module.
Work on the imgSeek-lite (http://imgseek.sourceforge.net/lite/ --
'imgseek-lite' cvs module) subproject will also slowdown, as most of
the code there will evolve into the 2nd imgseek-net client prototype.
Requires
--------
- Python 2.2.x, QT 3.x and PyQT 3.5. (3.4 should work)
- ImageMagick development files or QT development files.
Recommended:
- Python Imaging Library.
Links
-----
Download: http://sourceforge.net/project/showfiles.php?group_id=70373
Homepage: http://imgseek.sourceforge.net/
Screenshots: http://imgseek.sourceforge.net/sshot/
Complete ChangeLog: http://imgseek.sourceforge.net/changelog.html
Best regards,
---
rnc <nieder(a)mail.ru>
The Vision Egg is a powerful, flexible, and free way to produce
stimuli for vision research experiments available at:
http://www.visionegg.org/
The Vision Egg is a high level interface between Python and OpenGL. In
addition to methods for automatic generation of traditional visual
stimuli such as sinusoidal gratings and random dot patterns, it has a
number of functions for moving numeric data, images, movies, and text
to and from your video card. Therefore, it is also useful for anyone
wishing to make use of the features of today's graphics cards.
Features
========
* Perform experiments using an inexpensive PC and standard consumer
graphics card
* Perform experiments using a graphics workstation if special features
needed
* Data acquisition and other realtime hardware control capabilities
useful in electrophysiology and fMRI experiments, including
gaze-contingent stimuli
* Dynamically generated stimuli can be changed in realtime via
software or external hardware
* Produce traditional stimuli to replace legacy systems
* Produce stimuli not possible using other hardware
* Demo programs to get you started right away
* Run stimuli on your laptop - great for talks
* Free, open-source software
It runs on anything from cheap PCs to expensive special hardware for
special needs. For example, running on some platforms, such as SGI
workstations, the Vision Egg has a 10-bit luminance dynamic range
(both pixel depth and DAC) and precise frame-by-frame control.
The Vision Egg is open source software (GNU LGPL).
- Andrew Straw
Summary
A powerful and robust templating system for Python.
Overview
EmPy is a system for embedding Python expressions and statements
in template text; it takes an EmPy source file, processes it, and
produces output. This is accomplished via expansions, which are
special signals to the EmPy system and are set off by a special
prefix (by default the at sign, '@'). EmPy can expand arbitrary
Python expressions and statements in this way, as well as a
variety of special forms. Textual data not explicitly delimited
in this way is sent unaffected to the output, allowing Python to
be used in effect as a markup language. Also supported are
callbacks via hooks, recording and playback via diversions, and
dynamic, chainable filters. The system is highly configurable via
command line options and embedded commands.
Expressions are embedded in text with the '@(...)' notation;
variations include conditional expressions with '@(...?...!...)'
and the ability to handle thrown exceptions with '@(...$...)'. As
a shortcut, simple variables and expressions can be abbreviated as
'@variable', '@object.attribute', '@function(arguments)',
'@sequence[index]', and combinations. Full-fledged statements
are embedded with '@{...}'. Control flow in terms of conditional
or repeated expansion is available with '@[...]'. A '@' followed
by a whitespace character (including a newline) expands to
nothing, allowing string concatenations and line continuations.
Comments are indicated with '@#' and consume the rest of the line,
up to and including the trailing newline. '@%' indicate
"significators," which are special forms of variable assignment
intended to specify per-file identification information in a
format which is easy to parse externally. Context name and line
number changes can be done with '@?' and '@!' respectively.
Escape sequences analogous to those in C can be specified with
'@\...', and finally a '@@' sequence expands to a single literal
at sign.
Getting the software
The current version of empy is 3.1.1.
The latest version of the software is available in a tarball here:
http://www.alcyone.com/software/empy/empy-latest.tar.gz.
The official URL for this Web site is
http://www.alcyone.com/software/empy/.
Requirements
EmPy should work with any version of Python from 1.5.2 onward. It
has been tested with all major versions of CPython from 1.5 up,
and Jython from 2.0 up (using Java runtimes 1.3 and 1.4). The
included test script is intended to run on Unix-like systems with
a Bourne shell.
License
This code is released under the GPL.
....
Release history (since 3.1)
- 3.1.1; 2003 Sep 20. Added literal '@"..."' markup; added
--pause-at-end command line option; fixed improper globals
collision error via the 'sys.stdout' proxy.