REST API delete user function does not delete every 2nd linked address
(you’ll need to widen your message window to see this clearly)
The following series of curl commands creates a user with an email address of ‘z@example.org’, and then links 10 additional addresses to z@example.org
curl -X POST --data "email=z@example.org&display_name=z" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users curl -X POST --data "email=z1@example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org/addresses curl -X POST --data "email=z2@example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org/addresses curl -X POST --data "email=z3@example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org/addresses curl -X POST --data "email=z4@example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org/addresses curl -X POST --data "email=z5@example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org/addresses curl -X POST --data "email=z6@example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org/addresses curl -X POST --data "email=z7@example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org/addresses curl -X POST --data "email=z8@example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org/addresses curl -X POST --data "email=z9@example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org/addresses curl -X POST --data "email=z10@example.org" --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org/addresses
A sqlite3 dump of the database shows the following in the addresses table:
INSERT INTO "address" VALUES(11,'z@example.org',NULL,'z',NULL,'2015-02-08 10:50:37.241724',5,17); INSERT INTO "address" VALUES(12,'z1@example.org',NULL,'',NULL,'2015-02-08 10:52:08.928901',5,19); INSERT INTO "address" VALUES(13,'z2@example.org',NULL,'',NULL,'2015-02-08 10:52:36.308978',5,20); INSERT INTO "address" VALUES(14,'z3@example.org',NULL,'',NULL,'2015-02-08 10:52:43.947229',5,21); INSERT INTO "address" VALUES(15,'z4@example.org',NULL,'',NULL,'2015-02-08 10:52:49.598934',5,22); INSERT INTO "address" VALUES(16,'z5@example.org',NULL,'',NULL,'2015-02-08 10:52:54.406450',5,23); INSERT INTO "address" VALUES(17,'z6@example.org',NULL,'',NULL,'2015-02-08 10:52:59.295078',5,24); INSERT INTO "address" VALUES(18,'z7@example.org',NULL,'',NULL,'2015-02-08 10:53:04.892381',5,25); INSERT INTO "address" VALUES(19,'z8@example.org',NULL,'',NULL,'2015-02-08 10:53:10.497119',5,26); INSERT INTO "address" VALUES(20,'z9@example.org',NULL,'',NULL,'2015-02-08 10:53:16.450507',5,27); INSERT INTO "address" VALUES(21,'z10@example.org',NULL,'',NULL,'2015-02-08 10:53:24.021368',5,28);
All good at this stage. No problems.
The following curl command deletes user ‘z@example.org’
curl --verbose -X DELETE --header "authorization: Basic cmVzdGFkbWluOnJlc3RwYXNz" http://localhost:8001/3.0/users/z@example.org
Strangely, a sqlite3 dump of the database now shows the following in the addresses table:
INSERT INTO "address" VALUES(12,'z1@example.org',NULL,'',NULL,'2015-02-08 10:52:08.928901',NULL,19); INSERT INTO "address" VALUES(14,'z3@example.org',NULL,'',NULL,'2015-02-08 10:52:43.947229',NULL,21); INSERT INTO "address" VALUES(16,'z5@example.org',NULL,'',NULL,'2015-02-08 10:52:54.406450',NULL,23); INSERT INTO "address" VALUES(18,'z7@example.org',NULL,'',NULL,'2015-02-08 10:53:04.892381',NULL,25); INSERT INTO "address" VALUES(20,'z9@example.org',NULL,'',NULL,'2015-02-08 10:53:16.450507',NULL,27);
So 6 of the addresses that should have been deleted were deleted, but 5 remain. The linked address deletion seems to be skipping addresses for some reason.
I might need some help to track this down……..
When a user is deleted via the REST API, there is a function that appears to be iterating over the users addresses and unlinking them.
Is it possible that the unlinking process is somehow mutating the list of addresses that calling function is using? So as each address is unlinked somehow the underlying list of addresses is no longer able to be used as an effective address iterator?
Here are the relevant functions:
in rest/users.py (this is call from the REST API)
def on_delete(self, request, response):
"""Delete the named user, all her memberships, and addresses."""
if self._user is None:
not_found(response)
return
for member in self._user.memberships.members:
member.unsubscribe()
user_manager = getUtility(IUserManager)
for address in self._user.addresses:
debug_log.info('address scheduled for deletion: {}'.format(address))
for address in self._user.addresses:
user_manager.delete_address(address)
user_manager.delete_user(self._user)
no_content(response)*********> the debug.log line above returns a full list of all the addresses scheduled for deletion, proving that all the expected addresses are present in self._user.addresses *********> THE NEXT LINE "for address in self._user.addresses:” IS DOING THE SKIPPING. *********>>> SO - COULD IT BE that self._user.addresses is being mutated by user_manager.delete_address(address), stuffing up the for loop that is meant to be iterating over all addresses? Which is to say, as each address is unlinked, the length of and values in self._user.addresses is changing?
in usermanager.py
@dbconnection
def delete_address(self, store, address):
"""See `IUserManager`."""
# If there's a user controlling this address, it has to first be
# unlinked before the address can be deleted.
if address.user:
address.user.unlink(address)
store.delete(address)This appears to fix the problme in rest/users.py
def on_delete(self, request, response):
"""Delete the named user, all her memberships, and addresses."""
if self._user is None:
not_found(response)
return
for member in self._user.memberships.members:
member.unsubscribe()
user_manager = getUtility(IUserManager)
addresses_for_deletion = []
for address in self._user.addresses:
# to avoid mutating the self._user.addresses iterator, create a separate list of addresses
addresses_for_deletion.append(address)
for address in addresses_for_deletion:
user_manager.delete_address(address)
user_manager.delete_user(self._user)
no_content(response)On 8 Feb 2015, at 11:07 pm, Andrew Stuart <andrew.stuart@supercoders.com.au> wrote:
I might need some help to track this down……..
When a user is deleted via the REST API, there is a function that appears to be iterating over the users addresses and unlinking them.
Is it possible that the unlinking process is somehow mutating the list of addresses that calling function is using? So as each address is unlinked somehow the underlying list of addresses is no longer able to be used as an effective address iterator?
Here are the relevant functions:
in rest/users.py (this is call from the REST API)
def on_delete(self, request, response): """Delete the named user, all her memberships, and addresses.""" if self._user is None: not_found(response) return for member in self._user.memberships.members: member.unsubscribe() user_manager = getUtility(IUserManager) for address in self._user.addresses: debug_log.info('address scheduled for deletion: {}'.format(address)) for address in self._user.addresses: user_manager.delete_address(address) user_manager.delete_user(self._user) no_content(response)
*********> the debug.log line above returns a full list of all the addresses scheduled for deletion, proving that all the expected addresses are present in self._user.addresses *********> THE NEXT LINE "for address in self._user.addresses:” IS DOING THE SKIPPING. *********>>> SO - COULD IT BE that self._user.addresses is being mutated by user_manager.delete_address(address), stuffing up the for loop that is meant to be iterating over all addresses? Which is to say, as each address is unlinked, the length of and values in self._user.addresses is changing?
in usermanager.py
@dbconnection
def delete_address(self, store, address):
"""See IUserManager."""
# If there's a user controlling this address, it has to first be
# unlinked before the address can be deleted.
if address.user:
address.user.unlink(address)
store.delete(address)
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40s...
Security Policy: http://wiki.list.org/x/QIA9
On Feb 09, 2015, at 07:06 AM, Andrew Stuart wrote:
Thanks for tracking this down. I don't know whether this may have changed between Storm and SQLAlchemy, but it's very likely at least now caused by the old mutate-dictionary-while-iterating problem. With the switch to Python 3, if it were a dictionary an exception would have been thrown:
Python 3.4.2 (default, Feb 3 2015, 14:26:38) [GCC 4.9.2] on linux Type "help", "copyright", "credits" or "license" for more information.
Thanks for opening the bug; I'll work up a test case and a fix, but I think you're in the right direction.
Cheers, -Barry
I put a suggested fix in the bug report.
as
On 9 Feb 2015, at 10:52 am, Barry Warsaw <barry@list.org> wrote:
On Feb 09, 2015, at 07:06 AM, Andrew Stuart wrote:
Thanks for tracking this down. I don't know whether this may have changed between Storm and SQLAlchemy, but it's very likely at least now caused by the old mutate-dictionary-while-iterating problem. With the switch to Python 3, if it were a dictionary an exception would have been thrown:
Python 3.4.2 (default, Feb 3 2015, 14:26:38) [GCC 4.9.2] on linux Type "help", "copyright", "credits" or "license" for more information.
Thanks for opening the bug; I'll work up a test case and a fix, but I think you're in the right direction.
Cheers, -Barry
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40s...
Security Policy: http://wiki.list.org/x/QIA9
I might need some help to track this down……..
When a user is deleted via the REST API, there is a function that appears to be iterating over the users addresses and unlinking them.
Is it possible that the unlinking process is somehow mutating the list of addresses that calling function is using? So as each address is unlinked somehow the underlying list of addresses is no longer able to be used as an effective address iterator?
Here are the relevant functions:
in rest/users.py (this is call from the REST API)
def on_delete(self, request, response):
"""Delete the named user, all her memberships, and addresses."""
if self._user is None:
not_found(response)
return
for member in self._user.memberships.members:
member.unsubscribe()
user_manager = getUtility(IUserManager)
for address in self._user.addresses:
debug_log.info('address scheduled for deletion: {}'.format(address))
for address in self._user.addresses:
user_manager.delete_address(address)
user_manager.delete_user(self._user)
no_content(response)*********> the debug.log line above returns a full list of all the addresses scheduled for deletion, proving that all the expected addresses are present in self._user.addresses *********> THE NEXT LINE "for address in self._user.addresses:” IS DOING THE SKIPPING. *********>>> SO - COULD IT BE that self._user.addresses is being mutated by user_manager.delete_address(address), stuffing up the for loop that is meant to be iterating over all addresses? Which is to say, as each address is unlinked, the length of and values in self._user.addresses is changing?
in usermanager.py
@dbconnection
def delete_address(self, store, address):
"""See `IUserManager`."""
# If there's a user controlling this address, it has to first be
# unlinked before the address can be deleted.
if address.user:
address.user.unlink(address)
store.delete(address)This appears to fix the problme in rest/users.py
def on_delete(self, request, response):
"""Delete the named user, all her memberships, and addresses."""
if self._user is None:
not_found(response)
return
for member in self._user.memberships.members:
member.unsubscribe()
user_manager = getUtility(IUserManager)
addresses_for_deletion = []
for address in self._user.addresses:
# to avoid mutating the self._user.addresses iterator, create a separate list of addresses
addresses_for_deletion.append(address)
for address in addresses_for_deletion:
user_manager.delete_address(address)
user_manager.delete_user(self._user)
no_content(response)On 8 Feb 2015, at 11:07 pm, Andrew Stuart <andrew.stuart@supercoders.com.au> wrote:
I might need some help to track this down……..
When a user is deleted via the REST API, there is a function that appears to be iterating over the users addresses and unlinking them.
Is it possible that the unlinking process is somehow mutating the list of addresses that calling function is using? So as each address is unlinked somehow the underlying list of addresses is no longer able to be used as an effective address iterator?
Here are the relevant functions:
in rest/users.py (this is call from the REST API)
def on_delete(self, request, response): """Delete the named user, all her memberships, and addresses.""" if self._user is None: not_found(response) return for member in self._user.memberships.members: member.unsubscribe() user_manager = getUtility(IUserManager) for address in self._user.addresses: debug_log.info('address scheduled for deletion: {}'.format(address)) for address in self._user.addresses: user_manager.delete_address(address) user_manager.delete_user(self._user) no_content(response)
*********> the debug.log line above returns a full list of all the addresses scheduled for deletion, proving that all the expected addresses are present in self._user.addresses *********> THE NEXT LINE "for address in self._user.addresses:” IS DOING THE SKIPPING. *********>>> SO - COULD IT BE that self._user.addresses is being mutated by user_manager.delete_address(address), stuffing up the for loop that is meant to be iterating over all addresses? Which is to say, as each address is unlinked, the length of and values in self._user.addresses is changing?
in usermanager.py
@dbconnection
def delete_address(self, store, address):
"""See IUserManager."""
# If there's a user controlling this address, it has to first be
# unlinked before the address can be deleted.
if address.user:
address.user.unlink(address)
store.delete(address)
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40s...
Security Policy: http://wiki.list.org/x/QIA9
On Feb 09, 2015, at 07:06 AM, Andrew Stuart wrote:
Thanks for tracking this down. I don't know whether this may have changed between Storm and SQLAlchemy, but it's very likely at least now caused by the old mutate-dictionary-while-iterating problem. With the switch to Python 3, if it were a dictionary an exception would have been thrown:
Python 3.4.2 (default, Feb 3 2015, 14:26:38) [GCC 4.9.2] on linux Type "help", "copyright", "credits" or "license" for more information.
Thanks for opening the bug; I'll work up a test case and a fix, but I think you're in the right direction.
Cheers, -Barry
I put a suggested fix in the bug report.
as
On 9 Feb 2015, at 10:52 am, Barry Warsaw <barry@list.org> wrote:
On Feb 09, 2015, at 07:06 AM, Andrew Stuart wrote:
Thanks for tracking this down. I don't know whether this may have changed between Storm and SQLAlchemy, but it's very likely at least now caused by the old mutate-dictionary-while-iterating problem. With the switch to Python 3, if it were a dictionary an exception would have been thrown:
Python 3.4.2 (default, Feb 3 2015, 14:26:38) [GCC 4.9.2] on linux Type "help", "copyright", "credits" or "license" for more information.
Thanks for opening the bug; I'll work up a test case and a fix, but I think you're in the right direction.
Cheers, -Barry
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40s...
Security Policy: http://wiki.list.org/x/QIA9
participants (2)
-
Andrew Stuart -
Barry Warsaw