[Bug 1236297] [NEW] Unicode errors with PostgreSQL and RawStr fields
Public bug reported: When using PostgreSQL as a database, it's impossible to set the moderator_password field of a mailing list, which is of type RawStr. To be precise, it's possible to set it, but getting it back from the database after a commit will cause a TypeError with the message: "Expected str, found <type 'unicode'>" as if PostgreSQL always returned unicode on TEXT fields and Storm did not convert them to bytestrings. I get no errors on an SQLite database. I don't know if the problem comes from Mailman or Storm. I have written a python script to reproduce it, see the attached file. There are comments, it should be easily readable. ** Affects: mailman Importance: Undecided Status: New ** Tags: mailman3 ** Attachment added: "test-mm-storm-unicode.py" https://bugs.launchpad.net/bugs/1236297/+attachment/3864008/+files/test-mm-s... -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1236297 Title: Unicode errors with PostgreSQL and RawStr fields To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1236297/+subscriptions
I found why! In "mailman/src/mailman/database/schema/postgres.sql", the moderator_password field is of type "TEXT" instead of being "BYTEA". As a result, the psycopg2 driver converts it to an unicode string. This is coherent with the Storm manual which states that RawStr types should be set as BYTEA types in PostgreSQL: https://storm.canonical.com/Manual#Table_of_properties_vs._python_vs._databa... I believe a schema migration script will be needed. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1236297 Title: Unicode errors with PostgreSQL and RawStr fields To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1236297/+subscriptions
Here's the schema migration script, it's pretty simple. I'm converting existing data to ASCII and not UTF-8 because we don't expect non-ascii chars to be in the moderator_password column (it should contain the result of the config.password_context.encrypt() function, which is ASCII-only. ** Attachment added: "mm_20131007000000.py" https://bugs.launchpad.net/mailman/+bug/1236297/+attachment/3864274/+files/m... -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1236297 Title: Unicode errors with PostgreSQL and RawStr fields To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1236297/+subscriptions
** Changed in: mailman Milestone: None => 3.0.0b4 ** Changed in: mailman Assignee: (unassigned) => Barry Warsaw (barry) ** Changed in: mailman Importance: Undecided => High ** Changed in: mailman Status: New => In Progress ** Branch linked: lp:~nkarageuzian/mailman/pg_update -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1236297 Title: Unicode errors with PostgreSQL and RawStr fields To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1236297/+subscriptions
** Changed in: mailman Milestone: 3.0.0b4 => 3.0.0b5 -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1236297 Title: Unicode errors with PostgreSQL and RawStr fields To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1236297/+subscriptions
This bug is no longer valid since Storm has been replaced by SQLAlchemy. ** Changed in: mailman Status: In Progress => Invalid -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1236297 Title: Unicode errors with PostgreSQL and RawStr fields To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1236297/+subscriptions
participants (2)
-
Aurélien Bompard -
Barry Warsaw