From tlocke at tlocke.org.uk Sat Nov 9 16:39:51 2013 From: tlocke at tlocke.org.uk (Tony Locke) Date: Sat, 9 Nov 2013 15:39:51 +0000 Subject: [DB-SIG] Iterable cursor as part of core rather than an extension in DBAPI 3 In-Reply-To: References: Message-ID: Thanks for your comments Daniele. I've abandoned the idea of getting rid of fetchall(), fetchmany() and fetchone(), but I've kept the idea of execute() returning self. I've implemented it in my fantasy DB-API 3 branch at https://github.com/tlocke/pg8000/tree/three. Cheers, Tony. On 25 October 2013 11:31, Daniele Varrazzo wrote: > On Thu, Oct 24, 2013 at 10:57 AM, Tony Locke wrote: > > Hi, in DBAPI 3 is there a plan to bring the 'iterable cursor' extension > into > > the core? Looking at https://wiki.python.org/moin/DbApi3 I couldn't see > > anything. The reason I ask is that I'm noticing on PG8000 that the time > > taken to execute the warn() for each call to next() is significant for > > iterating over a large number of rows. > > PG8000 is pretty much free to not raise the warnings, if it creates > problems and you report it as a bug to its developers. Of course the > warning could be also raises once with a lightweight check instead of > relying on the Python machinery to suppress duplicates, if it gets in > the way. I don't think the deficiencies in an adapter's implementation > should be driving the standard's design. > > > > I'm on the verge of suggesting getting rid of fetchmany() and just having > > the iterable, and making execute() return the cursor. Then instead of: > > > > cursor.execute("select * from emp") > > all_emps = cursor.fetchmany() > > > > you'd have: > > > > all_emps = [cursor.execute("select * from emp")] > > > > or you could do: > > > > for emp in cursor.execute("select * from emp"): > > pass > > From this example it seems you want to get rid of fetchall() instead. > Fetchmany has other useful use cases. In order to enable the idiom of > the example, if a cursor supports iteration, it is enough for > cursor.execute() to return "self": IIRC some adapters do that, and I > find it a clever idea. > > > -- Daniele > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlocke at tlocke.org.uk Sun Nov 10 21:49:35 2013 From: tlocke at tlocke.org.uk (Tony Locke) Date: Sun, 10 Nov 2013 20:49:35 +0000 Subject: [DB-SIG] asyncio support In-Reply-To: References: Message-ID: What sort of thing do you have in mind? Perhaps something where the execute() method returns immediately, and a callback function is called when the query is complete? This could be implemented in a library on top of DB-API. But should it be part of the spec? Cheers, Tony. On 25 October 2013 08:27, INADA Naoki wrote: > Does someone consider DB-API for asyncio? > > > -- > INADA Naoki > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Mon Nov 11 10:43:38 2013 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 11 Nov 2013 18:43:38 +0900 Subject: [DB-SIG] asyncio support In-Reply-To: References:

Message-ID: I think having definition about what method returns Future is good. For example: con = yield from pymysql.connect_async(...) # connection may cause cur = con.cursor() # cursor creation should done without asyncio yield from cur.execute("...") # execute may use asyncio row = yield from cur.fetchone() # fetch also uses asyncio for row in cur: # raises NotImplementedError On Mon, Nov 11, 2013 at 5:49 AM, Tony Locke wrote: > What sort of thing do you have in mind? Perhaps something where the > execute() method returns immediately, and a callback function is called > when the query is complete? > > This could be implemented in a library on top of DB-API. But should it be > part of the spec? > > Cheers, > > Tony. > > > On 25 October 2013 08:27, INADA Naoki wrote: > >> Does someone consider DB-API for asyncio? >> >> >> -- >> INADA Naoki >> >> _______________________________________________ >> DB-SIG maillist - DB-SIG at python.org >> https://mail.python.org/mailman/listinfo/db-sig >> >> > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > > -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Nov 12 15:51:14 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 12 Nov 2013 15:51:14 +0100 Subject: [DB-SIG] asyncio support In-Reply-To: References:

<528205BE.8000204@python.org> Message-ID: <52824062.3040805@egenix.com> On 12.11.2013 12:03, INADA Naoki wrote: > On Tue, Nov 12, 2013 at 7:41 PM, M.-A. Lemburg wrote: > >> On 11.11.2013 10:43, INADA Naoki wrote: >>> I think having definition about what method returns Future is good. >>> >>> For example: >>> >>> con = yield from pymysql.connect_async(...) # connection may cause >>> cur = con.cursor() # cursor creation should done without asyncio >>> yield from cur.execute("...") # execute may use asyncio >>> row = yield from cur.fetchone() # fetch also uses asyncio >>> >>> for row in cur: # raises NotImplementedError >> >> This can already be implemented using e.g. gevent, provided the >> client side is implemented in Python (and gevent can thus patch >> the socket module). >> > > I don't talk about "How to use connection with asyncio?" > > I'm one of committer of PyMySQL and I want to support asyncio. > > And I think standard api for supporting asyncio is nice. With asyncio you mean the new tulip support in Python 3.4 ? We could add something as standard extension of the DB-API, but not (yet) to the standard itself, since the DB-API still has to support Python 2.x. However, before coming up with a standard extension, I'd suggest to first get some idea of what the asyncio best practices are. Ideally, it should also be possible to implement asynchronous .execute() and .fetch...() without using the asyncio module and in Python versions prior to Python 3.4. >> For native C client libs, this will not easily be possible, since >> the Python wrapper will typically not get access to the underlying >> socket logic, so you have to rely on native async support in the >> library. >> >> E.g. ODBC has limited support for this, but uses an active >> polling mechanism, so callbacks won't work and I'm not >> sure Futures would directly. You would have to hook up the >> polling with the async event loop. >> > > Yes. So async API should be optional. > > >> >> What always works is using threads as a way to let other >> code continue to run. .execute() and .fetch...() usually release >> the GIL for this purpose. >> > > I want to make PyMySQL works with asyncio in single thread. > > Does any developers of DB connectors concern about asyncio? I'm not sure I understand the question. I've looked into adding async support to our native mxODBC interface, but the polling mechanism used by the ODBC API has made this somewhat complicated. It is also not clear how expensive such a polling call would be and whether the ODBC drivers support for async I/O is mature enough yet (it took around 5-10 years for them to properly support Unicode, so I don't have great expectations). Other client libs may implement callbacks for async processing or not support it at all. If we come up with a standard, it would have to support asyncio (ie. having methods return Futures), gevent, callbacks and polling, I guess. For mxODBC Connect, we've implemented gevent compatibility, so you can use async processing without having to change your code - which IMHO is the nicest way to do async processing :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 12 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-11-19: Python Meeting Duesseldorf ... 7 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Tue Nov 12 16:14:05 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Tue, 12 Nov 2013 10:14:05 -0500 Subject: [DB-SIG] asyncio support In-Reply-To: <52824062.3040805@egenix.com> References:

<528205BE.8000204@python.org> <52824062.3040805@egenix.com> Message-ID: <7FE2493C-7943-48B3-8B44-0707A2FEC855@zzzcomputing.com> On Nov 12, 2013, at 9:51 AM, M.-A. Lemburg wrote: > > For mxODBC Connect, we've implemented gevent compatibility, > so you can use async processing without having to change > your code - which IMHO is the nicest way to do async > processing :-) though sadly just doesn?t seem to be catching on, as the rise of systems like Node.js continue to spread the idea that you need to wrangle explicit callback spaghetti, descending fully and explicitly throughout all libraries and system calls, in order to achieve asynchronous nirvana, whatever that means. Using gevent means you don?t get to have the satisfaction of admiring your newly forged, self-satisfying maze of callbacks - it makes programming practical and boring all over again. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at python.org Tue Nov 12 11:41:02 2013 From: mal at python.org (M.-A. Lemburg) Date: Tue, 12 Nov 2013 11:41:02 +0100 Subject: [DB-SIG] asyncio support In-Reply-To: References:

Message-ID: <528205BE.8000204@python.org> On 11.11.2013 10:43, INADA Naoki wrote: > I think having definition about what method returns Future is good. > > For example: > > con = yield from pymysql.connect_async(...) # connection may cause > cur = con.cursor() # cursor creation should done without asyncio > yield from cur.execute("...") # execute may use asyncio > row = yield from cur.fetchone() # fetch also uses asyncio > > for row in cur: # raises NotImplementedError This can already be implemented using e.g. gevent, provided the client side is implemented in Python (and gevent can thus patch the socket module). For native C client libs, this will not easily be possible, since the Python wrapper will typically not get access to the underlying socket logic, so you have to rely on native async support in the library. E.g. ODBC has limited support for this, but uses an active polling mechanism, so callbacks won't work and I'm not sure Futures would directly. You would have to hook up the polling with the async event loop. What always works is using threads as a way to let other code continue to run. .execute() and .fetch...() usually release the GIL for this purpose. > On Mon, Nov 11, 2013 at 5:49 AM, Tony Locke wrote: > >> What sort of thing do you have in mind? Perhaps something where the >> execute() method returns immediately, and a callback function is called >> when the query is complete? >> >> This could be implemented in a library on top of DB-API. But should it be >> part of the spec? >> >> Cheers, >> >> Tony. >> >> >> On 25 October 2013 08:27, INADA Naoki wrote: >> >>> Does someone consider DB-API for asyncio? >>> >>> >>> -- >>> INADA Naoki >>> >>> _______________________________________________ >>> DB-SIG maillist - DB-SIG at python.org >>> https://mail.python.org/mailman/listinfo/db-sig >>> >>> >> >> _______________________________________________ >> DB-SIG maillist - DB-SIG at python.org >> https://mail.python.org/mailman/listinfo/db-sig >> >> > > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > -- Marc-Andre Lemburg Director Python Software Foundation http://www.python.org/psf/ From songofacandy at gmail.com Tue Nov 12 12:03:12 2013 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 12 Nov 2013 20:03:12 +0900 Subject: [DB-SIG] asyncio support In-Reply-To: <528205BE.8000204@python.org> References:

<528205BE.8000204@python.org> Message-ID: On Tue, Nov 12, 2013 at 7:41 PM, M.-A. Lemburg wrote: > On 11.11.2013 10:43, INADA Naoki wrote: > > I think having definition about what method returns Future is good. > > > > For example: > > > > con = yield from pymysql.connect_async(...) # connection may cause > > cur = con.cursor() # cursor creation should done without asyncio > > yield from cur.execute("...") # execute may use asyncio > > row = yield from cur.fetchone() # fetch also uses asyncio > > > > for row in cur: # raises NotImplementedError > > This can already be implemented using e.g. gevent, provided the > client side is implemented in Python (and gevent can thus patch > the socket module). > I don't talk about "How to use connection with asyncio?" I'm one of committer of PyMySQL and I want to support asyncio. And I think standard api for supporting asyncio is nice. > > For native C client libs, this will not easily be possible, since > the Python wrapper will typically not get access to the underlying > socket logic, so you have to rely on native async support in the > library. > > E.g. ODBC has limited support for this, but uses an active > polling mechanism, so callbacks won't work and I'm not > sure Futures would directly. You would have to hook up the > polling with the async event loop. > Yes. So async API should be optional. > > What always works is using threads as a way to let other > code continue to run. .execute() and .fetch...() usually release > the GIL for this purpose. > I want to make PyMySQL works with asyncio in single thread. Does any developers of DB connectors concern about asyncio? -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From john at nmt.edu Wed Nov 20 02:27:28 2013 From: john at nmt.edu (John W. Shipman) Date: Tue, 19 Nov 2013 18:27:28 -0700 (MST) Subject: [DB-SIG] Oracle+SQLA: Where are my tables? Message-ID: I'm using SQLAlchemy and cx_Oracle to explore a legacy Oracle database. When I use Oracle's sqldeveloper application, I can see the tables and their contents just fine. When I connect to it using SQLAlchemy, I can create the engine with no problem, but after I call the .reflect() method on the MetaData instance, its .sorted_tables list is empty. If I connect to a Postgres database and do the same thing, the .sorted_tables list is as the documentation says it should be. Any suggestions on how to find the tables? A coworker says that tables in Oracle are segregated into namespaces. Is that my problem? I would greatly appreciate any assistance. Best regards, John Shipman, Web Developer, National Radio Astronomy Observatories; john at nmt.edu, http://www.nmt.edu/~shipman/ From mike_mp at zzzcomputing.com Wed Nov 20 16:08:01 2013 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Wed, 20 Nov 2013 10:08:01 -0500 Subject: [DB-SIG] Oracle+SQLA: Where are my tables? In-Reply-To: References: Message-ID: For SQLAlchemy questions, come on over to our list at https://groups.google.com/forum/#!forum/sqlalchemy . as far as your .reflect() method, your tables might be under a different owner than the user who you are connecting as, and they may or may not appear to be ?visible? locally if the system includes synonyms which match their names. The ?schema? argument to MetaData.reflect() would allow this owner name to be specified. To diagnose the problem, set echo=?debug? on create_engine() as it reflects the tables, then look to evaluate the results being returned by the system view queries being emitted by the reflection process. On Nov 19, 2013, at 8:27 PM, John W. Shipman wrote: > I'm using SQLAlchemy and cx_Oracle to explore a legacy Oracle > database. When I use Oracle's sqldeveloper application, I can > see the tables and their contents just fine. > > When I connect to it using SQLAlchemy, I can create the engine > with no problem, but after I call the .reflect() method on the > MetaData instance, its .sorted_tables list is empty. If I > connect to a Postgres database and do the same thing, the > .sorted_tables list is as the documentation says it should be. > > Any suggestions on how to find the tables? A coworker says that > tables in Oracle are segregated into namespaces. Is that my > problem? > > I would greatly appreciate any assistance. > > Best regards, > John Shipman, Web Developer, National Radio Astronomy Observatories; > john at nmt.edu, http://www.nmt.edu/~shipman/ > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From john at nmt.edu Wed Nov 20 18:46:59 2013 From: john at nmt.edu (John W. Shipman) Date: Wed, 20 Nov 2013 10:46:59 -0700 (MST) Subject: [DB-SIG] Oracle+SQLA: Where are my tables? In-Reply-To: References:

Message-ID: On Wed, 20 Nov 2013, Michael Bayer wrote: +-- | For SQLAlchemy questions, come on over to our list at | https://groups.google.com/forum/#!forum/sqlalchemy . +-- Thanks, I'll bookmark that! Your suggestion of using echo='debug' got me through the logjam. My coworker came up with the right value for the schema argument to .reflect(), and I can see the tables now. Here are the relevant lines that worked: ================================================================ eng=engine.create_engine(dsn) meta=schema.MetaData(bind=eng) meta.reflect(schema='foo') ---------------------------------------------------------------- The meta.tables.keys() values have the form u'foo.TABLE', but that's no problem. Thank you very much! If you hadn't helped me, I wouldn't have known where to turn next. With gratitude, John Shipman, Web Developer, National Radio Astronomy Observatories; john at nmt.edu, http://www.nmt.edu/~shipman/