compiling psycopg2 with zc.recipe.egg: allowing custom build_ext options?
Regarding Chris recent thread on doing some hair pulling while getting psycopg2 to install, I recently added this distribution to a buildout with the following configuration: [versions] pscopg2 = 2.0.8 [buildout] parts = app postgresql init-pgsql pgsql-symlinks psycopg2 find-links = http://initd.org/pub/software/psycopg/PSYCOPG-2-0/ versions = versions # app is a Grok-based application that requires pscopg2 [app] recipe = zc.recipe.egg eggs = ${psycopg2:egg} gsc.docsdb z3c.evalexception>=2.0 Paste PasteScript PasteDeploy interpreter = python-console # PostgreSQL parts [psycopg2] recipe = zc.recipe.egg:custom egg = psycopg2 include-dirs = ${postgresql:location}/include library-dirs = ${postgresql:location}/lib [postgresql] recipe = zc.recipe.cmmi url = http://wwwmaster.postgresql.org/redir/198/h/source/v8.3.7/postgresql-8.3.7.t... extra_options = --with-readline [init-pgsql] recipe = iw.recipe.cmd on_install = true on_update = false cmds = ${postgresql:location}/bin/initdb -D ${postgresql:location}/var/data -E UNICODE [pgsql-symlinks] recipe = cns.recipe.symlink symlink_target = ${buildout:directory}/bin symlink_base = ${postgresql:location}/bin symlink = clusterdb createdb createlang createuser dropdb droplang dropuser ecpg initdb ipcclean pg_config pg_controldata pg_ctl pg_dump pg_dumpall pg_resetxlog pg_restore postgres postmaster psql reindexdb vacuumdb This builds a complete PostgreSQL install inside the project directory. Of course, if you've already got a preferred postgres install, you can simplify this quite a bit and just point the include- dirs and library-dirs to that location. I did find that I had to pin psycopg2 down to 2.0.8, since it didn't seem to work with 2.0.10. But looking a bit further into this issue, the preferred way to build psycopg2 is to supply the full path to pg_config as a build_ext. However, zc.recipe.egg:custom only supports a fixed list of build_ext options. Is this a feature or a bug? e.g. Is there any reason why it wouldn't be possible to support arbitrary build_ext options with this recipe? Or is there another egg-building recipe out there which supports this? By patching a local copy of zc.recipe.egg to allow the pg_config build_ext option, I was able to use this part to compile psycopg2: [psycopg2] recipe = zc.recipe.egg:custom egg = psycopg2 pg_config = ${buildout:bin-directory}/pg_config Which worked fine for the latest release, 2.0.10, of psycopg2. (also, as a bonus 'Arg!', note that it's necessary to supply a custom find-links URL to buildout, since the PyPI page for psycopg2 has bogus download links)
Here is how i would integrate postgresql in my python based projects: My 2 cents ;) ========================================== minitage installation. ========================================== virtualenv --no-site-packages distutils source distutils/bin/activate cd distutils easy_install -U minitage.core && minimerge -s ========================================================= Installing psycopg2 without a dedicated minitage project ========================================================= Install postgresql --------------------- minimerge postgresql-8.4 mkdir pgsql pushd pgsql cat << EOF > buildout.cfg [versions] psycopg2 = 2.0.12 [buildout] versions=versions extensions=buildout.minitagificator parts = psycopg [minitage] dependencies = postgresql-8.4 [psycopg] recipe=minitage.recipe.scripts eggs=psycopg2 interpreter=zopepy # use python datetime! # if you are inside a virtualenv but the python from which you forked has # mx inside, but you dont want to have mx.datetime inside your local virtualenv # you will need to add this bits to avoid build problems psycopg2-bdistext-use_pydatetime=1 psycopg2-bdistext-mx_include_dir=NONEXISTINGDIR EOF buildout bootstrap bin/buildout Checks --------------- ./foo/myproject/bin/zopepy -c "import psycopg2;print psycopg2.__version__" ... 2.0.12 (dt dec ext pq3) ... grep psycopg bin/zopepy ... sys.path[0:0] = [ '/home/kiorky/tmp/distutils/pgsql/eggs/psycopg2-2.0.12-py2.6-linux-i686.egg' ] ... # replace by otool -L on mac ldd eggs/psycopg2-2*.egg/psycopg2/*so ... libpq.so.5 => /home/kiorky/tmp/distutils/dependencies/postgresql-8.4/parts/part/lib/libpq.so.5 (0xb7ecd000) ... libssl.so.0.9.8 => /home/kiorky/tmp/distutils/dependencies/postgresql-8.4/../openssl-0.9/parts/part/lib/libssl.so.0.9.8 (0xb7d02000) libcrypto.so.0.9.8 => /home/kiorky/tmp/distutils/dependencies/postgresql-8.4/../openssl-0.9/parts/part/lib/libcrypto.so.0.9.8 (0xb7bab000) ... popd ==================================================================================== Installing psycopg2 inside a minitagified project (something that have a minibuild) ==================================================================================== Create the minilay/minibuild ----------------------------------- mkdir -p minilays/myprojectminilay/ cat << EOF > minilays/myprojectminilay/myproject [minibuild] dependencies=postgresql-8.4 src_uri="http://foo" # note that if you archive that somewhere, it would we reusable src_type=static category=foo install_method=buildout EOF Create the project dir ------------------------ Note that it is only needed the first time and if you archived your project, you ll just have to checkout the minilay and do "minimerge myproject" mkdir -p foo/myproject/ cat << EOF > foo/myproject/buildout.cfg [versions] psycopg2 = 2.0.12 [buildout] versions=versions extensions=buildout.minitagificator eggs-directory=../../eggs/cache parts = psycopg [psycopg] recipe=minitage.recipe.scripts eggs=psycopg2 interpreter=zopepy # use python datetime! # if you are inside a virtualenv but the python from which you forked has # mx inside, but you dont want to have mx.datetime inside your local virtualenv # you will need to add this bits to avoid build problems psycopg2-bdistext-use_pydatetime=1 psycopg2-bdistext-mx_include_dir=NONEXISTINGDIR EOF Install it ------------ minimerge -NRv myproject Checks --------- grep psy foo/myproject/bin/zopepy ... sys.path[0:0] = [ '/home/kiorky/tmp/distutils/eggs/cache/psycopg2-2.0.12-py2.6-linux-i686.egg' ] ... ./foo/myproject/bin/zopepy -c "import psycopg2;print psycopg2.__version__" 2.0.12 (dt dec ext pq3) # Note that the difference there is that we have include all needed dependencies thanks to the minibuilds infos. ldd ../eggs/cache//psycopg2-2*.egg/psycopg2/*so ... linux-gate.so.1 => (0xffffe000) libpython2.6.so.1.0 => /home/kiorky/tmp/distutils/dependencies/python-2.6/parts/part/lib/libpython2.6.so.1.0 (0xb7e61000) libpq.so.5 => /home/kiorky/tmp/distutils/dependencies/postgresql-8.4/parts/part/lib/libpq.so.5 (0xb7e3d000) libpthread.so.0 => /lib/libpthread.so.0 (0xb7dfc000) libc.so.6 => /lib/libc.so.6 (0xb7cbf000) libz.so.1 => /home/kiorky/tmp/distutils/dependencies/zlib-1.2/parts/part/lib/libz.so.1 (0xb7ca9000) libcrypt.so.1 => /lib/libcrypt.so.1 (0xb7c77000) libbz2.so.1.0 => /home/kiorky/tmp/distutils/dependencies/bzip2-1.0/parts/part/lib/libbz2.so.1.0 (0xb7c65000) libpanel.so.5 => /home/kiorky/tmp/distutils/dependencies/ncurses-5.6/parts/part/lib/libpanel.so.5 (0xb7c5f000) libform.so.5 => /home/kiorky/tmp/distutils/dependencies/ncurses-5.6/parts/part/lib/libform.so.5 (0xb7c50000) libmenu.so.5 => /home/kiorky/tmp/distutils/dependencies/ncurses-5.6/parts/part/lib/libmenu.so.5 (0xb7c47000) libncurses.so.5 => /home/kiorky/tmp/distutils/dependencies/ncurses-5.6/parts/part/lib/libncurses.so.5 (0xb7bfa000) libhistory.so.5 => /home/kiorky/tmp/distutils/dependencies/readline-5.2/parts/part/lib/libhistory.so.5 (0xb7bef000) libreadline.so.5 => /home/kiorky/tmp/distutils/dependencies/readline-5.2/parts/part/lib/libreadline.so.5 (0xb7bb8000) libcrypto.so.0.9.8 => /home/kiorky/tmp/distutils/dependencies/openssl-0.9/parts/part/lib/libcrypto.so.0.9.8 (0xb7a62000) libssl.so.0.9.8 => /home/kiorky/tmp/distutils/dependencies/openssl-0.9/parts/part/lib/libssl.so.0.9.8 (0xb7a1b000) libexpat.so.1 => /home/kiorky/tmp/distutils/dependencies/expat-2.0/parts/part/lib/libexpat.so.1 (0xb79ef000) libdb-4.7.so => /home/kiorky/tmp/distutils/dependencies/db-4.7/parts/part/lib/libdb-4.7.so (0xb7841000) libdl.so.2 => /lib/libdl.so.2 (0xb783d000) libutil.so.1 => /lib/libutil.so.1 (0xb7839000) libm.so.6 => /lib/libm.so.6 (0xb7813000) /lib/ld-linux.so.2 (0xb8079000) ... Install the "env" instance to have symlinks to minitage binaries ------------------------------------------------------------------ easy_install -U minitage.paste paster create --no-interactive -t minitage.instances.env myproject Use it ------- source foo/myproject/sys/share/minitage/minitage.env which psql ... /home/kiorky/tmp/distutils/dependencies/postgresql-8.4/parts/part/bin/psql ... You can even create a postgresql instance for your project --------------------------------------------------------------- Be ware to shutdown your running pgsql instance or run in interative and change the port: paster create --no-interactive -t minitage.instances.postgresql myproject ... Installation is now finished. * You can look for wrappers to various postgresql scripts located in "/home/kiorky/tmp/distutils/foo/myproject/sys/bin/myproject.*" eg : myproject.psql. You must use them as they are configured to use some useful defaults to connect to your database. * A configuration file for your postgresql instance has been linked from /home/kiorky/tmp/distutils/foo/myproject/sys/var/data/postgresql/myproject/postgresql.conf to /home/kiorky/tmp/distutils/foo/myproject/sys/etc/postgresql/myproject_myproject.postgresql.conf. * A pg_hba file for your postgresql instance has been linked from /home/kiorky/tmp/distutils/foo/myproject/sys/var/data/postgresql/myproject/pg_hba.conf to /home/kiorky/tmp/distutils/foo/myproject/sys/etc/postgresql/myproject_myproject.pg_hba.conf. * A pg_ident file for your postgresql instance has been linked from /home/kiorky/tmp/distutils/foo/myproject/sys/var/data/postgresql/myproject/pg_ident.conf to /home/kiorky/tmp/distutils/foo/myproject/sys/etc/postgresql/myproject_myproject.pg_ident.conf. * A init script to start your server is available in /home/kiorky/tmp/distutils/foo/myproject/sys/etc/init.d/postgresql_myproject.myproject. * A logrotate configuration file to handle your logs can be linked in global scope, it is available in /home/kiorky/tmp/distutils/foo/myproject/sys/etc/logrotate.d/myproject_myproject.postgresql. * By default, the user who created the database (kiorky) is now also superuser on it, only via localhost connections or via socket. * By default, you can connect to your database with the user 'myproject' and the supplied password Please note that you are also trusted on localhost. * For security rezasons, PostGreSQL only listens on localhost, change it in the configuration file if you want it to listen to other adresses. * The datadir is located under /home/kiorky/tmp/distutils/foo/myproject/sys/var/data/postgresql/myproject. * You can use pypgoptimizator to Tune automaticly your postgresql: easy_install pypgoptimizator pypgoptimizator -i /home/kiorky/tmp/distutils/foo/myproject/sys/var/data/postgresql/myproject/postgresql.conf -o /home/kiorky/tmp/distutils/foo/myproject/sys/var/data/postgresql/myproject/postgresql.conf ... Launching the server ----------------------------- source foo/myproject/sys/share/minitage/minitage.env ... foo/myproject/sys/etc/init.d/postgresql_myproject.myproject restart * Restarting PostgreSQL ... * Stopping PostgreSQL ...pg_ctl: PID file "/home/kiorky/tmp/distutils/foo/myproject/sys/var/data/postgresql/myproject/postmaster.pid" does not exist Is server running? [ !! ] * ... * Starting PostgreSQL ... ... Connecting ----------------- myproject.psql ... myproject.psql psql (8.4.0) Type "help" for help. myproject=> \l List of databases Name | Owner | Encoding | Collation | Ctype | Access privileges -----------+-----------+----------+-----------+-------+---------------------------- myproject | myproject | UTF8 | C | C | =Tc/myproject : myproject=C*T*c*/myproject postgres | kiorky | UTF8 | C | C | template0 | kiorky | UTF8 | C | C | =c/kiorky : kiorky=CTc/kiorky template1 | kiorky | UTF8 | C | C | =c/kiorky : kiorky=CTc/kiorky (4 rows) myproject=> ... Kevin Teague a écrit :
Regarding Chris recent thread on doing some hair pulling while getting psycopg2 to install, I recently added this distribution to a buildout with the following configuration:
[versions] pscopg2 = 2.0.8
[buildout] parts = app postgresql init-pgsql pgsql-symlinks psycopg2 find-links = http://initd.org/pub/software/psycopg/PSYCOPG-2-0/ versions = versions
# app is a Grok-based application that requires pscopg2 [app] recipe = zc.recipe.egg eggs = ${psycopg2:egg} gsc.docsdb z3c.evalexception>=2.0 Paste PasteScript PasteDeploy interpreter = python-console
# PostgreSQL parts
[psycopg2] recipe = zc.recipe.egg:custom egg = psycopg2 include-dirs = ${postgresql:location}/include library-dirs = ${postgresql:location}/lib
[postgresql] recipe = zc.recipe.cmmi url = http://wwwmaster.postgresql.org/redir/198/h/source/v8.3.7/postgresql-8.3.7.t... extra_options = --with-readline
[init-pgsql] recipe = iw.recipe.cmd on_install = true on_update = false cmds = ${postgresql:location}/bin/initdb -D ${postgresql:location}/var/data -E UNICODE
[pgsql-symlinks] recipe = cns.recipe.symlink symlink_target = ${buildout:directory}/bin symlink_base = ${postgresql:location}/bin symlink = clusterdb createdb createlang createuser dropdb droplang dropuser ecpg initdb ipcclean pg_config pg_controldata pg_ctl pg_dump pg_dumpall pg_resetxlog pg_restore postgres postmaster psql reindexdb vacuumdb
This builds a complete PostgreSQL install inside the project directory. Of course, if you've already got a preferred postgres install, you can simplify this quite a bit and just point the include- dirs and library-dirs to that location.
I did find that I had to pin psycopg2 down to 2.0.8, since it didn't seem to work with 2.0.10. But looking a bit further into this issue, the preferred way to build psycopg2 is to supply the full path to pg_config as a build_ext.
However, zc.recipe.egg:custom only supports a fixed list of build_ext options. Is this a feature or a bug? e.g. Is there any reason why it wouldn't be possible to support arbitrary build_ext options with this recipe? Or is there another egg-building recipe out there which supports this?
By patching a local copy of zc.recipe.egg to allow the pg_config build_ext option, I was able to use this part to compile psycopg2:
[psycopg2] recipe = zc.recipe.egg:custom egg = psycopg2 pg_config = ${buildout:bin-directory}/pg_config
Which worked fine for the latest release, 2.0.10, of psycopg2.
(also, as a bonus 'Arg!', note that it's necessary to supply a custom find-links URL to buildout, since the PyPI page for psycopg2 has bogus download links)
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF
participants (2)
-
Kevin Teague -
kiorky