The last few weeks I've been thinking about the architectural pattern known as Clean, Onion, Hexagonal, or Ports'n'Adaptors <http://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html> (http://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html). I'm curious if many people are applying it in the Django world? I haven't, yet, but I'm thinking of refactoring a vertical slice of our monster Django app into this style, and I'd love to hear if you think it's crazy / brilliant / obvious / old-hat, etc. Jonathan -- Jonathan Hartley tartley@tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley
I go a little way in this direction. I rarely go as far as dependency injection for business logic, but a good habit I picked up is to ensure that my business logic is put in appropriate places somewhere other than view functions, and all queries are kept with the models (ie. as Manager methods), and all the presentation operations are template tags and filters. That way the view contains little more than just connecting queries to business logic to templates, which is useful when it comes to understanding how an application was plumbed. In fact I've just used this to quite rapidly port a Postgres backed Django app to entirely ElasticSearch backed. Because I had defined my queries in terms of Manager methods it wasn't hard to reimplement them one-by-one with equivalent ElasticSearch searches. It would have been much harder if I'd littered .filter()/.exclude() calls all over the place. Something that was useful for that and slightly mitigates Django in this regard is that the lazily-evaluated, sliceable QuerySet has become something of a pattern, to the extent that you can sometimes sort of plug QuerySet-like result objects, such as those from PyES, MongoEngine, couchdb-python etc, into logic built around querysets. It also helps that Django templates are very forgiving. While interviewing for Python developers I found numerous candidates who when presented with a generic OOP problem would immediately start writing Django models, with the result that the solution exhibits exactly the kind of dependency problems that article describes. For similar reasons, we now have an application where the cryptically terse dictionary keys favoured by hardcore MongoDB geeks leak everywhere, including template code and client-side javascript.
On 2012-12-04 14:46, Jonathan Hartley wrote:
I haven't, yet, but I'm thinking of refactoring a vertical slice of our monster Django app into this style, and I'd love to hear if you think it's crazy / brilliant / obvious / old-hat, etc.
Since you mentioned this a few weeks back, I've been thinking about this approach a lot and doing a fair bit of background reading on the idea. I think it falls in to the brilliantly-obvious category, at least for any app of significant size. I plan to start using these ideas for my next big work project (not Django however). Previously, I've tended towards less flexible, harder-to-test, layered architectures. My main concern is that I can see this approach resulting in a lot of extra code to support the interfaces between the core application and the peripheral components. This is probably worth the cost however. I guess I'll see once I use the approach in anger. Menno
On 04/12/2012 17:46, Menno Smits wrote:
On 2012-12-04 14:46, Jonathan Hartley wrote:
I haven't, yet, but I'm thinking of refactoring a vertical slice of our monster Django app into this style, and I'd love to hear if you think it's crazy / brilliant / obvious / old-hat, etc.
Since you mentioned this a few weeks back, I've been thinking about this approach a lot and doing a fair bit of background reading on the idea. I think it falls in to the brilliantly-obvious category, at least for any app of significant size. I plan to start using these ideas for my next big work project (not Django however). Previously, I've tended towards less flexible, harder-to-test, layered architectures.
The biggest concern I have with this approach is that it appears to preclude taking advantage of any features of your storage layer. If your business objects have to not know or care about their underlying storage, how do you take advantage of nice relational queries, stand alone text indexing service or other specific features of the architecture you've chosen? There's also the risk that you end up testing each bit (business, views, storage) in isolation and never get around to doing the automated integration testing required to ensure artifacts of implementation (*cough* bugs) don't cause problems across those boundaries... That said, I'd love to see a project that's a good example of this, are there any around online in Python? The closest thing I can think of is the move to the component architecture that Zope did from v2 to v3; architecturally brilliant but actually killed off the platform... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On 5 Dec 2012, at 07:33, Chris Withers <chris@python.org> wrote:
On 04/12/2012 17:46, Menno Smits wrote:
On 2012-12-04 14:46, Jonathan Hartley wrote:
I haven't, yet, but I'm thinking of refactoring a vertical slice of our monster Django app into this style, and I'd love to hear if you think it's crazy / brilliant / obvious / old-hat, etc.
Since you mentioned this a few weeks back, I've been thinking about this approach a lot and doing a fair bit of background reading on the idea. I think it falls in to the brilliantly-obvious category, at least for any app of significant size. I plan to start using these ideas for my next big work project (not Django however). Previously, I've tended towards less flexible, harder-to-test, layered architectures.
The biggest concern I have with this approach is that it appears to preclude taking advantage of any features of your storage layer. If your business objects have to not know or care about their underlying storage, how do you take advantage of nice relational queries, stand alone text indexing service or other specific features of the architecture you've chosen?
I guess you still need to provide an abstraction for these features, even if only one backend supports the abstraction.
There's also the risk that you end up testing each bit (business, views, storage) in isolation and never get around to doing the automated integration testing required to ensure artifacts of implementation (*cough* bugs) don't cause problems across those boundaries...
Well, you need integration / acceptance / functional / end to end tests *anyway*. Michael
That said, I'd love to see a project that's a good example of this, are there any around online in Python? The closest thing I can think of is the move to the component architecture that Zope did from v2 to v3; architecturally brilliant but actually killed off the platform...
cheers,
Chris
-- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On 5 Dec 2012, at 07:33, Chris Withers <chris@python.org> wrote:
On 04/12/2012 17:46, Menno Smits wrote:
On 2012-12-04 14:46, Jonathan Hartley wrote:
I haven't, yet, but I'm thinking of refactoring a vertical slice of our monster Django app into this style, and I'd love to hear if you think it's crazy / brilliant / obvious / old-hat, etc. Since you mentioned this a few weeks back, I've been thinking about this approach a lot and doing a fair bit of background reading on the idea. I think it falls in to the brilliantly-obvious category, at least for any app of significant size. I plan to start using these ideas for my next big work project (not Django however). Previously, I've tended towards less flexible, harder-to-test, layered architectures. The biggest concern I have with this approach is that it appears to preclude taking advantage of any features of your storage layer. If your business objects have to not know or care about their underlying storage, how do you take advantage of nice relational queries, stand alone text indexing service or other specific features of the architecture you've chosen? I guess you still need to provide an abstraction for these features, even if only one backend supports the abstraction. What I'm reading suggests that if, for example, your app needs to use a text indexing service, then that's an external system and your application core should define an API which it will use to talk to it. So you still get to use features like this, but you are obliged to put an API layer (and possible inversion of control) between them, rather
On 05/12/2012 13:08, Michael Foord wrote: than making the call directly from within your business logic. Similarly, if your business logic needs a small number of particular relational-type queries, then perhaps those queries should be the API defined by your core for talking to persistence. If your app needs many such queries, or general-purpose relational querying support, then perhaps you can create an API which supports general purpose querying such as the 'repository' pattern. I'm no expert. The above is my newfound understanding and some speculation.
There's also the risk that you end up testing each bit (business, views, storage) in isolation and never get around to doing the automated integration testing required to ensure artifacts of implementation (*cough* bugs) don't cause problems across those boundaries... Well, you need integration / acceptance / functional / end to end tests *anyway*.
Michael
That said, I'd love to see a project that's a good example of this, are there any around online in Python? The closest thing I can think of is the move to the component architecture that Zope did from v2 to v3; architecturally brilliant but actually killed off the platform...
cheers,
Chris
-- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
-- http://www.voidspace.org.uk/
May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
_______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
-- Jonathan Hartley tartley@tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley
On 05/12/2012 14:13, Jonathan Hartley wrote:
I guess you still need to provide an abstraction for these features, even if only one backend supports the abstraction. What I'm reading suggests that if, for example, your app needs to use a text indexing service, then that's an external system and your application core should define an API which it will use to talk to it.
Isn't this just the API provided by whatever python library you choose to use?
So you still get to use features like this, but you are obliged to put an API layer (and possible inversion of control) between them, rather than making the call directly from within your business logic.
What's the actual benefit of this other than an additional layer of complexity and one more call in the stack?
Similarly, if your business logic needs a small number of particular relational-type queries, then perhaps those queries should be the API defined by your core for talking to persistence.
This would appear to kill the ad hoc querying that makes relational databases so powerful. sqlalchemy is about as much of an abstraction as I'd ever want...
If your app needs many such queries, or general-purpose relational querying support, then perhaps you can create an API which supports general purpose querying such as the 'repository' pattern.
Again, what's the actual benefit of this other than the theoretical ability to switch implementations, which you'll likely never use? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On 5 Dec 2012, at 07:33, Chris Withers wrote:
That said, I'd love to see a project that's a good example of this, are there any around online in Python? The closest thing I can think of is the move to the component architecture that Zope did from v2 to v3; architecturally brilliant but actually killed off the platform…
Or more correctly, architecturally brilliant so much so that it was taken forth to be used outside the platform to such a degree that the rest of the platform (Zope 3) was refactored out as a bunch of standalone libraries (The Zope Toolkit). To the OP: The Zope Component Architecture may cover quite a bit of what you want to do… take a look at zope.component and zope.interface (or read up here http://bluebream.zope.org/doc/1.0/manual/componentarchitecture.html) -Matt -- Matt Hamilton, Technical Director Netsight Internet Solutions Limited http://www.netsight.co.uk/matth Tel: 0117 90 90 90 1 Ext. 15 Registered in England No. 3892180 Registered office: 40 Berkeley Square, Clifton, Bristol, BS8 1HU
On 5 December 2012 07:33, Chris Withers <chris@python.org> wrote:
The closest thing I can think of is the move to the component architecture that Zope did from v2 to v3; architecturally brilliant but actually killed off the platform...
ROTFL! I have to admit my first thought on reading that page and diagram was "architecture astronauts"... If you're building a system which will last many years and tie together a core business, AND it really needs a rich internal object model embodying "business rules", yes, maybe this approach is valid. A core ERP system or some kinds of financial modelling systems spring to mind. Basically, a project which can justify a core team of 3-6 people working on it long term. But if you really need a company database, and a variety of web front ends and interfaces to it including CRUD, and budgets/timescales are limited, then there are huge efficiency savings all round for trying to do things "by the book" in Django, and using models as your business objects. New developers know where to look for things, and a lot of so-called business rules are easily implemented by save methods and signals. And there are established best practices for testing. This week we had to help an overseas firm add a reporting capability to their Django app. They sent over a copy of the code, and their development practices were amazingly similar to ours, and we had it running in an hour or so. Everything was where you expected to find it. If we had needed to trace through 3-4 layers to understand where one factoid came from, life would be a lot harder. I do fully agree with having an agreed dependency graph. We will often centralize the 'non-Django' parts of an application as helper functions inside a python module, inputting and returning just primitive Python tests, and write unit tests for those. But that's more a matter of having a well written "utils.py" in an app than of reimplementing everything outside of your framework. Cooperating web services have also helped deal with the interfaces problem. Big hairy corporate systems are often broken down now into reasonable sized chunks that POST and GET to each other, so you don't need quite so many layers and adapters in the code of any one of them. Just my grumpy-old-man 2p worth. -- Andy Robinson
On Tue, 4 Dec 2012, Jonathan Hartley wrote:
The last few weeks I've been thinking about the architectural pattern known as Clean, Onion, Hexagonal, or Ports'n'Adaptors <http://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html> (http://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html). I'm curious if many people are applying it in the Django world?
I haven't, yet, but I'm thinking of refactoring a vertical slice of our monster Django app into this style, and I'd love to hear if you think it's crazy / brilliant / obvious / old-hat, etc.
I have to confess I've only very lightly skimmed the article (which looks like it says some sensible things), but that's not going to stop me pontificating in over-general terms and posting a video that I liked: The best programmers I've worked with have a knack to ruthlessly pick the simplest possible abstractions to fit the job in hand. They never stop thinking to settle on any one-size-fits-all programming style. The problem with those two statements I just made is that everybody can read them and think that they agree with them. What *I* mean by simple is close to what Rich Hickey means in the first part of this talk (though I don't know enough to decide what I think about how he goes on to defend Clojure and its design principles in those terms): http://www.infoq.com/presentations/Simple-Made-Easy (BTW, it's a shame to hear him give security as an example of a separable concern, because it isn't one) The best code, you look at the functionality, then the code, and think "where is all the code?" and "how did such simplistic code happen to implement exactly what was needed?". That's different from the "OMG, what is all this stuff for" feeling you get from over-engineered or just badly-factored code. The best code is easy to change in the sense that changes in functionality require commensurate coding effort, and it's clear what code would have to change. But it is also hard to change, in the sense that any change that leaves the behaviour the same would clearly make it worse -- including adding or removing abstraction. I'd agree with Andy that not fighting too many battles with your framework has a lot to be said for it (and that it's maybe more important to nail basic coding practices of the kind you'd find in Code Complete than to take even a single step away from what the django tutorial tells you to do). But even short programs can gain simplicity from ignoring the framework or abstracting it a little, where it suits the problem, as it often does. John
Yeah, what he said :-) (joking aside, John has summed this all up very nicely...) Chris On 06/12/2012 00:57, John Lee wrote:
On Tue, 4 Dec 2012, Jonathan Hartley wrote:
The last few weeks I've been thinking about the architectural pattern known as Clean, Onion, Hexagonal, or Ports'n'Adaptors <http://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html> (http://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html). I'm curious if many people are applying it in the Django world?
I haven't, yet, but I'm thinking of refactoring a vertical slice of our monster Django app into this style, and I'd love to hear if you think it's crazy / brilliant / obvious / old-hat, etc.
I have to confess I've only very lightly skimmed the article (which looks like it says some sensible things), but that's not going to stop me pontificating in over-general terms and posting a video that I liked:
The best programmers I've worked with have a knack to ruthlessly pick the simplest possible abstractions to fit the job in hand. They never stop thinking to settle on any one-size-fits-all programming style. The problem with those two statements I just made is that everybody can read them and think that they agree with them. What *I* mean by simple is close to what Rich Hickey means in the first part of this talk (though I don't know enough to decide what I think about how he goes on to defend Clojure and its design principles in those terms):
http://www.infoq.com/presentations/Simple-Made-Easy
(BTW, it's a shame to hear him give security as an example of a separable concern, because it isn't one)
The best code, you look at the functionality, then the code, and think "where is all the code?" and "how did such simplistic code happen to implement exactly what was needed?". That's different from the "OMG, what is all this stuff for" feeling you get from over-engineered or just badly-factored code. The best code is easy to change in the sense that changes in functionality require commensurate coding effort, and it's clear what code would have to change. But it is also hard to change, in the sense that any change that leaves the behaviour the same would clearly make it worse -- including adding or removing abstraction.
I'd agree with Andy that not fighting too many battles with your framework has a lot to be said for it (and that it's maybe more important to nail basic coding practices of the kind you'd find in Code Complete than to take even a single step away from what the django tutorial tells you to do). But even short programs can gain simplicity from ignoring the framework or abstracting it a little, where it suits the problem, as it often does.
John _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________
-- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Thanks everyone. I've been enlightened, encouraged and forewarned by the responses to my initial question. (and thanks for the Rich Hickey talk, I hadn't seen that one) I absolutely empathise with the warnings to avoid needlessly creating extra layers or plumbing. I have a colleague who says many of the projects where he used to work at Microsoft Research used this pattern, and while he liked it overall, if he had to find fault, they found it to cause duplication of sorts, in things like the persistence component mapping fields of the business objects into database-compatible types. Hence adding a field meant both changing the core business object, and changing the persistence component to incorporate the extra field, and also the plumbing between the two. I also heed the advice to beware shoehorning problems into a single one-size-fits-all solution. I also liked people who brought up particular thorny areas with Django in particular, such as the admin views. However, I'm hopeful that, in my particular case at least, the interfaces and extra layers will be close to trivial, and any disadvantages may be outweighed by extracting my database/network code from my business logic. I'm starting my week-long exercise on trying it out on a slice of our existing Django monster. I'll report back. Jonathan On 10/12/2012 19:02, Chris Withers wrote:
Yeah, what he said :-)
(joking aside, John has summed this all up very nicely...)
Chris
On 06/12/2012 00:57, John Lee wrote:
On Tue, 4 Dec 2012, Jonathan Hartley wrote:
The last few weeks I've been thinking about the architectural pattern known as Clean, Onion, Hexagonal, or Ports'n'Adaptors <http://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html>
(http://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html).
I'm curious if many people are applying it in the Django world?
I haven't, yet, but I'm thinking of refactoring a vertical slice of our monster Django app into this style, and I'd love to hear if you think it's crazy / brilliant / obvious / old-hat, etc.
I have to confess I've only very lightly skimmed the article (which looks like it says some sensible things), but that's not going to stop me pontificating in over-general terms and posting a video that I liked:
The best programmers I've worked with have a knack to ruthlessly pick the simplest possible abstractions to fit the job in hand. They never stop thinking to settle on any one-size-fits-all programming style. The problem with those two statements I just made is that everybody can read them and think that they agree with them. What *I* mean by simple is close to what Rich Hickey means in the first part of this talk (though I don't know enough to decide what I think about how he goes on to defend Clojure and its design principles in those terms):
http://www.infoq.com/presentations/Simple-Made-Easy
(BTW, it's a shame to hear him give security as an example of a separable concern, because it isn't one)
The best code, you look at the functionality, then the code, and think "where is all the code?" and "how did such simplistic code happen to implement exactly what was needed?". That's different from the "OMG, what is all this stuff for" feeling you get from over-engineered or just badly-factored code. The best code is easy to change in the sense that changes in functionality require commensurate coding effort, and it's clear what code would have to change. But it is also hard to change, in the sense that any change that leaves the behaviour the same would clearly make it worse -- including adding or removing abstraction.
I'd agree with Andy that not fighting too many battles with your framework has a lot to be said for it (and that it's maybe more important to nail basic coding practices of the kind you'd find in Code Complete than to take even a single step away from what the django tutorial tells you to do). But even short programs can gain simplicity from ignoring the framework or abstracting it a little, where it suits the problem, as it often does.
John _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________
-- Jonathan Hartley tartley@tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley
I'm starting my week-long exercise on trying it out on a slice of our existing Django monster. I'll report back.
Echoes of Captain Oates here? Anyway, reviving an old thread: Brandon Rhodes talks provides a straightforward and accessible summary of "The Clean Architecture" in a recent talk[0] and it's the least wanky discussion of this kind that I've come across. His main point is that things with side effects (particularly calling external services) should be isolated in "procedural glue" code, while the processing of data -- the business logic -- should happen in pure functional code. This leads to a better focus on the structure of the data you're dealing with, which in turn leads to more comprehensible and maintainable code. Thoughts? [0] http://www.pyvideo.org/video/2840/the-clean-architecture-in-python On 13 December 2012 11:28, Jonathan Hartley <tartley@tartley.com> wrote:
Thanks everyone. I've been enlightened, encouraged and forewarned by the responses to my initial question. (and thanks for the Rich Hickey talk, I hadn't seen that one)
I absolutely empathise with the warnings to avoid needlessly creating extra layers or plumbing.
I have a colleague who says many of the projects where he used to work at Microsoft Research used this pattern, and while he liked it overall, if he had to find fault, they found it to cause duplication of sorts, in things like the persistence component mapping fields of the business objects into database-compatible types. Hence adding a field meant both changing the core business object, and changing the persistence component to incorporate the extra field, and also the plumbing between the two.
I also heed the advice to beware shoehorning problems into a single one-size-fits-all solution.
I also liked people who brought up particular thorny areas with Django in particular, such as the admin views.
However, I'm hopeful that, in my particular case at least, the interfaces and extra layers will be close to trivial, and any disadvantages may be outweighed by extracting my database/network code from my business logic.
I'm starting my week-long exercise on trying it out on a slice of our existing Django monster. I'll report back.
Jonathan
On 10/12/2012 19:02, Chris Withers wrote:
Yeah, what he said :-)
(joking aside, John has summed this all up very nicely...)
Chris
On 06/12/2012 00:57, John Lee wrote:
On Tue, 4 Dec 2012, Jonathan Hartley wrote:
The last few weeks I've been thinking about the architectural pattern
known as Clean, Onion, Hexagonal, or Ports'n'Adaptors <http://blog.8thlight.com/uncle-bob/2012/08/13/the- clean-architecture.html> (http://blog.8thlight.com/uncle-bob/2012/08/13/the- clean-architecture.html). I'm curious if many people are applying it in the Django world?
I haven't, yet, but I'm thinking of refactoring a vertical slice of our monster Django app into this style, and I'd love to hear if you think it's crazy / brilliant / obvious / old-hat, etc.
I have to confess I've only very lightly skimmed the article (which looks like it says some sensible things), but that's not going to stop me pontificating in over-general terms and posting a video that I liked:
The best programmers I've worked with have a knack to ruthlessly pick the simplest possible abstractions to fit the job in hand. They never stop thinking to settle on any one-size-fits-all programming style. The problem with those two statements I just made is that everybody can read them and think that they agree with them. What *I* mean by simple is close to what Rich Hickey means in the first part of this talk (though I don't know enough to decide what I think about how he goes on to defend Clojure and its design principles in those terms):
http://www.infoq.com/presentations/Simple-Made-Easy
(BTW, it's a shame to hear him give security as an example of a separable concern, because it isn't one)
The best code, you look at the functionality, then the code, and think "where is all the code?" and "how did such simplistic code happen to implement exactly what was needed?". That's different from the "OMG, what is all this stuff for" feeling you get from over-engineered or just badly-factored code. The best code is easy to change in the sense that changes in functionality require commensurate coding effort, and it's clear what code would have to change. But it is also hard to change, in the sense that any change that leaves the behaviour the same would clearly make it worse -- including adding or removing abstraction.
I'd agree with Andy that not fighting too many battles with your framework has a lot to be said for it (and that it's maybe more important to nail basic coding practices of the kind you'd find in Code Complete than to take even a single step away from what the django tutorial tells you to do). But even short programs can gain simplicity from ignoring the framework or abstracting it a little, where it suits the problem, as it often does.
John _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________
-- Jonathan Hartley tartley@tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley
_______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
On 13 August 2014 08:17, Peter Inglesby <peter.inglesby@gmail.com> wrote:
I'm starting my week-long exercise on trying it out on a slice of our existing Django monster. I'll report back.
Echoes of Captain Oates here?
Anyway, reviving an old thread: Brandon Rhodes talks provides a straightforward and accessible summary of "The Clean Architecture" in a recent talk[0] and it's the least wanky discussion of this kind that I've come across. His main point is that things with side effects (particularly calling external services) should be isolated in "procedural glue" code, while the processing of data -- the business logic -- should happen in pure functional code. This leads to a better focus on the structure of the data you're dealing with, which in turn leads to more comprehensible and maintainable code.
Thoughts?
[0] http://www.pyvideo.org/video/2840/the-clean-architecture-in-python
I'm always reminded of that Rimmer quote about Oates... I've been doing this recently with Twisted -- separating out async code that deals with deferred chains from synchronous logic handling the results of the deferreds. I'm finding that it's leading to clear, easily tested code -- simple synchronous tests on real data for the business logic, and stubbed success/failures to test the various code-paths in the deferred logic. I came to this variously by tangling too much with mixes of deferred-flows and business logic while trying to write simple tests and seeing this DAS talk on Boundaries [1] d = defer.succeed(foo) d.addCallback(handle_foo) d.addCallback(make_request) d.addErrback(retry, make_request, foo) d.addCallback(parse_body) return d ^ code like this is easily tested by stubbing all the callbacks, with the stubs either succeeding or raising synchronous exceptions. def parse_body(body): content = json.parse(body.read()) if not 'username' in content: raise HttpError(401) return Response(content) ^ again, really easy to test. No deep nested logic, no need to mock or stub. [1] https://www.destroyallsoftware.com/talks/boundaries
The Brandon Rhodes talk sounds like a nice way of looking at the problem. What I've done with my latest BIG DJANGO APP is to put the business logic into a pure Python module, Which Django imports, initialises with a storage adaptor (which makes mocking this adaptor easy) and then uses. The storage adaptor gives the module only Python primitives and the module only returns Python primitives. Which Django can then do whatever it wants with. Mostly creating http responses, but also putting them into Celery queues and the like. It's still a bit messy, but it's very testable. Hansel On Wed, Aug 13, 2014 at 11:32 AM, James Broadhead <jamesbroadhead@gmail.com> wrote:
On 13 August 2014 08:17, Peter Inglesby <peter.inglesby@gmail.com> wrote:
I'm starting my week-long exercise on trying it out on a slice of our existing Django monster. I'll report back.
Echoes of Captain Oates here?
Anyway, reviving an old thread: Brandon Rhodes talks provides a straightforward and accessible summary of "The Clean Architecture" in a recent talk[0] and it's the least wanky discussion of this kind that I've come across. His main point is that things with side effects (particularly calling external services) should be isolated in "procedural glue" code, while the processing of data -- the business logic -- should happen in pure functional code. This leads to a better focus on the structure of the data you're dealing with, which in turn leads to more comprehensible and maintainable code.
Thoughts?
[0] http://www.pyvideo.org/video/2840/the-clean-architecture-in-python
I'm always reminded of that Rimmer quote about Oates...
I've been doing this recently with Twisted -- separating out async code that deals with deferred chains from synchronous logic handling the results of the deferreds. I'm finding that it's leading to clear, easily tested code -- simple synchronous tests on real data for the business logic, and stubbed success/failures to test the various code-paths in the deferred logic. I came to this variously by tangling too much with mixes of deferred-flows and business logic while trying to write simple tests and seeing this DAS talk on Boundaries [1]
d = defer.succeed(foo) d.addCallback(handle_foo) d.addCallback(make_request) d.addErrback(retry, make_request, foo) d.addCallback(parse_body) return d ^ code like this is easily tested by stubbing all the callbacks, with the stubs either succeeding or raising synchronous exceptions.
def parse_body(body): content = json.parse(body.read()) if not 'username' in content: raise HttpError(401) return Response(content) ^ again, really easy to test. No deep nested logic, no need to mock or stub.
[1] https://www.destroyallsoftware.com/talks/boundaries
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
-- Hansel
Thanks for the Brandon talk link: I hadn't seen it yet and I'm eager to see anything new from him.
Oates
I did embark on the Django refactor, and in my mind it was a success of sorts. Obviously we didn't derive any immediate business value from that refactor, but it did demonstrate that the approach was viable, and I liked the way that the code turned out. I would seriously consider it on future projects (I'm not doing Django at the moment, and my current project has a large codebase that I'm not going to try to re-architect.) There were some sticking points that with hindsight seemed predictable. I don't know that I can remember them specifically now, but the pattern seemed to be that it would have been much easier if our code had kept Django at a bit more of arms-length, rather than wholeheartedly relying on it for *everything*. For example, we had used things like Django's magic 'settings' module to store all sorts of project-wide constants. This caused problems for the 'clean architecture' approach, because it meant that any code which wanted to access any settings needed to import the Django settings mechanism. Better, instead, to use Django's settings module to just store Django configuration options, and store project-wide constants in more generic place of your own devising. So, as advertised, it seemed that the 'clean' approach really did make it more visible when we weren't being very smart about managing our dependencies, and force us to limit them more than we previously had. Jonathan On 13/08/14 11:32, James Broadhead wrote:
On 13 August 2014 08:17, Peter Inglesby <peter.inglesby@gmail.com> wrote:
I'm starting my week-long exercise on trying it out on a slice of our existing Django monster. I'll report back.
Echoes of Captain Oates here?
Anyway, reviving an old thread: Brandon Rhodes talks provides a straightforward and accessible summary of "The Clean Architecture" in a recent talk[0] and it's the least wanky discussion of this kind that I've come across. His main point is that things with side effects (particularly calling external services) should be isolated in "procedural glue" code, while the processing of data -- the business logic -- should happen in pure functional code. This leads to a better focus on the structure of the data you're dealing with, which in turn leads to more comprehensible and maintainable code.
Thoughts?
[0] http://www.pyvideo.org/video/2840/the-clean-architecture-in-python
I'm always reminded of that Rimmer quote about Oates...
I've been doing this recently with Twisted -- separating out async code that deals with deferred chains from synchronous logic handling the results of the deferreds. I'm finding that it's leading to clear, easily tested code -- simple synchronous tests on real data for the business logic, and stubbed success/failures to test the various code-paths in the deferred logic. I came to this variously by tangling too much with mixes of deferred-flows and business logic while trying to write simple tests and seeing this DAS talk on Boundaries [1]
d = defer.succeed(foo) d.addCallback(handle_foo) d.addCallback(make_request) d.addErrback(retry, make_request, foo) d.addCallback(parse_body) return d ^ code like this is easily tested by stubbing all the callbacks, with the stubs either succeeding or raising synchronous exceptions.
def parse_body(body): content = json.parse(body.read()) if not 'username' in content: raise HttpError(401) return Response(content) ^ again, really easy to test. No deep nested logic, no need to mock or stub.
[1] https://www.destroyallsoftware.com/talks/boundaries
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
-- Jonathan Hartley tartley@tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley
Coincidentally, I blogged on the topic of Django project organisation at the weekend. http://mauveweb.co.uk/posts/2014/08/organising-django-projects.html May be of interest?
Thanks Peter! I was speaking to Brandon at Pycon this year and he was telling me this was going to be his next talk to take on the road, and I was definitely looking forward to seeing it. Matt O'Donnell was also there, and he's done a talk on this sort of thing recently too ( https://www.youtube.com/watch?v=NGhL7IA6Dik). It's definitely in the air. My own modest attempts to approach the subject are in my book -- in chapter 19, where I show how striving for test isolation can (theoretically) push you towards something like a lean architecture ( http://chimera.labs.oreilly.com/books/1234000000754/ch19.html) and in chapter 21, the wrap-up, where I waffle on about all these things ( http://chimera.labs.oreilly.com/books/1234000000754/ch22.html) I don't think I managed to broach the subject nearly as cleanly as Brandon did. I really admire his talks. His data structures talk was one of the top 3 I saw at Pycon this year ( http://pyvideo.org/video/2571/all-your-ducks-in-a-row-data-structures-in-the...). Perfect pace, slides that complement rather than repeat the talk, fascinating and useful content... Anyways, back to our onions - I guess the thing that's always bothered me a bit about the "clean architecture" is that my main project (pythonanywhere) is "all boundaries", to use Gary Bernhardt's terminology. Or, to put it differently, I don't think we really have much in the way of "business logic". We just turn Http requests into commands that go to processes. There's really not much in the way of "logic" in the way. No calculations or business rules to speak of. So it's never seemed worth it, to us. And sometimes I think -- aren't many web projects just thin CRUD wrappers around a database? Is going to all the trouble of isolating your business logic from, eg, django, really worth it in most cases? On 13 August 2014 13:09, Daniel Pope <lord.mauve@gmail.com> wrote:
Coincidentally, I blogged on the topic of Django project organisation at the weekend.
http://mauveweb.co.uk/posts/2014/08/organising-django-projects.html
May be of interest?
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
-- ------------------------------ Harry J.W. Percival ------------------------------ Twitter: @hjwp Mobile: +44 (0) 78877 02511 Skype: harry.percival
You know, if you started writing a stand alone module called pythonanywhere that has no dependencies and was imported and used by Django, Tornado, and any of the other bits and pieces I think you might find a *few *things to put in there... Not that I have any knowledge of that code base, of course ;-) On Fri, Aug 15, 2014 at 11:22 AM, Harry Percival <harry.percival@gmail.com> wrote:
Thanks Peter! I was speaking to Brandon at Pycon this year and he was telling me this was going to be his next talk to take on the road, and I was definitely looking forward to seeing it. Matt O'Donnell was also there, and he's done a talk on this sort of thing recently too ( https://www.youtube.com/watch?v=NGhL7IA6Dik). It's definitely in the air.
My own modest attempts to approach the subject are in my book -- in chapter 19, where I show how striving for test isolation can (theoretically) push you towards something like a lean architecture ( http://chimera.labs.oreilly.com/books/1234000000754/ch19.html) and in chapter 21, the wrap-up, where I waffle on about all these things ( http://chimera.labs.oreilly.com/books/1234000000754/ch22.html)
I don't think I managed to broach the subject nearly as cleanly as Brandon did. I really admire his talks. His data structures talk was one of the top 3 I saw at Pycon this year ( http://pyvideo.org/video/2571/all-your-ducks-in-a-row-data-structures-in-the...). Perfect pace, slides that complement rather than repeat the talk, fascinating and useful content...
Anyways, back to our onions - I guess the thing that's always bothered me a bit about the "clean architecture" is that my main project (pythonanywhere) is "all boundaries", to use Gary Bernhardt's terminology. Or, to put it differently, I don't think we really have much in the way of "business logic". We just turn Http requests into commands that go to processes. There's really not much in the way of "logic" in the way. No calculations or business rules to speak of. So it's never seemed worth it, to us.
And sometimes I think -- aren't many web projects just thin CRUD wrappers around a database? Is going to all the trouble of isolating your business logic from, eg, django, really worth it in most cases?
On 13 August 2014 13:09, Daniel Pope <lord.mauve@gmail.com> wrote:
Coincidentally, I blogged on the topic of Django project organisation at the weekend.
http://mauveweb.co.uk/posts/2014/08/organising-django-projects.html
May be of interest?
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
-- ------------------------------ Harry J.W. Percival ------------------------------ Twitter: @hjwp Mobile: +44 (0) 78877 02511 Skype: harry.percival
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
-- Hansel
I think what you describe is a common situation. When I reorganised that application in a Django project, that was one aspect that bugged some of my coworkers. But I think there is potentially still some value there. The resulting one line business functions create a well-defined seam between your app's (slender) business logic and each of your interfaces to external systems. That way, you make it easy for developers to avoid accidentally mixing 'django' code into the same module as code which talks to ElasticSearch. If you already have the discipline to maintain great separation of concerns (along these or other lines), which I'm guessing PythonAnywhere still does, then perhaps you don't need this particular constraint to help you maintain it. Is there also value in helping you plug in fake external services for testing purposes? I believe there is, but again, I'm not sure this value is greater than zero if you *already* have well thought-out mechanisms for plugging in fake external systems. Jonathan On 15/08/14 11:22, Harry Percival wrote:
Thanks Peter! I was speaking to Brandon at Pycon this year and he was telling me this was going to be his next talk to take on the road, and I was definitely looking forward to seeing it. Matt O'Donnell was also there, and he's done a talk on this sort of thing recently too ( https://www.youtube.com/watch?v=NGhL7IA6Dik). It's definitely in the air.
My own modest attempts to approach the subject are in my book -- in chapter 19, where I show how striving for test isolation can (theoretically) push you towards something like a lean architecture ( http://chimera.labs.oreilly.com/books/1234000000754/ch19.html) and in chapter 21, the wrap-up, where I waffle on about all these things ( http://chimera.labs.oreilly.com/books/1234000000754/ch22.html)
I don't think I managed to broach the subject nearly as cleanly as Brandon did. I really admire his talks. His data structures talk was one of the top 3 I saw at Pycon this year ( http://pyvideo.org/video/2571/all-your-ducks-in-a-row-data-structures-in-the...). Perfect pace, slides that complement rather than repeat the talk, fascinating and useful content...
Anyways, back to our onions - I guess the thing that's always bothered me a bit about the "clean architecture" is that my main project (pythonanywhere) is "all boundaries", to use Gary Bernhardt's terminology. Or, to put it differently, I don't think we really have much in the way of "business logic". We just turn Http requests into commands that go to processes. There's really not much in the way of "logic" in the way. No calculations or business rules to speak of. So it's never seemed worth it, to us.
And sometimes I think -- aren't many web projects just thin CRUD wrappers around a database? Is going to all the trouble of isolating your business logic from, eg, django, really worth it in most cases?
On 13 August 2014 13:09, Daniel Pope <lord.mauve@gmail.com> wrote:
Coincidentally, I blogged on the topic of Django project organisation at the weekend.
http://mauveweb.co.uk/posts/2014/08/organising-django-projects.html
May be of interest?
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
-- Jonathan Hartley tartley@tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley
This may be getting tenuous, but here's another example of what I think of as a "no-business-logic" app, and how we test-drove it: http://www.obeythetestinggoat.com/test-driving-a-docker-based-postgres-servi... Part of my blockage may come from the words "business logic" -- we don't have much in the way of "business rules" here, so maybe Brandon's definition -- business logic is all the stuff that can be expressed in purely functional terms, with no side-effects -- could help me... But even using that, so far all I've got is that we generate some paths on disks by concatenating some strings together with some constants, so that's really clutching at straws... On 15 August 2014 17:01, Jonathan Hartley <tartley@tartley.com> wrote:
I think what you describe is a common situation. When I reorganised that application in a Django project, that was one aspect that bugged some of my coworkers.
But I think there is potentially still some value there. The resulting one line business functions create a well-defined seam between your app's (slender) business logic and each of your interfaces to external systems. That way, you make it easy for developers to avoid accidentally mixing 'django' code into the same module as code which talks to ElasticSearch.
If you already have the discipline to maintain great separation of concerns (along these or other lines), which I'm guessing PythonAnywhere still does, then perhaps you don't need this particular constraint to help you maintain it.
Is there also value in helping you plug in fake external services for testing purposes? I believe there is, but again, I'm not sure this value is greater than zero if you *already* have well thought-out mechanisms for plugging in fake external systems.
Jonathan
On 15/08/14 11:22, Harry Percival wrote:
Thanks Peter! I was speaking to Brandon at Pycon this year and he was telling me this was going to be his next talk to take on the road, and I was definitely looking forward to seeing it. Matt O'Donnell was also there, and he's done a talk on this sort of thing recently too (https://www.youtube.com/watch?v=NGhL7IA6Dik). It's definitely in the air.
My own modest attempts to approach the subject are in my book -- in chapter 19, where I show how striving for test isolation can (theoretically) push you towards something like a lean architecture (http://chimera.labs.oreilly.com/books/1234000000754/ch19.html) and in chapter 21, the wrap-up, where I waffle on about all these things (http://chimera.labs.oreilly.com/books/1234000000754/ch22.html)
I don't think I managed to broach the subject nearly as cleanly as Brandon did. I really admire his talks. His data structures talk was one of the top 3 I saw at Pycon this year (http://pyvideo.org/video/2571/all-your-ducks-in-a-row-data-structures-in-the...). Perfect pace, slides that complement rather than repeat the talk, fascinating and useful content...
Anyways, back to our onions - I guess the thing that's always bothered me a bit about the "clean architecture" is that my main project (pythonanywhere) is "all boundaries", to use Gary Bernhardt's terminology. Or, to put it differently, I don't think we really have much in the way of "business logic". We just turn Http requests into commands that go to processes. There's really not much in the way of "logic" in the way. No calculations or business rules to speak of. So it's never seemed worth it, to us.
And sometimes I think -- aren't many web projects just thin CRUD wrappers around a database? Is going to all the trouble of isolating your business logic from, eg, django, really worth it in most cases?
On 13 August 2014 13:09, Daniel Pope <lord.mauve@gmail.com> <lord.mauve@gmail.com> wrote:
Coincidentally, I blogged on the topic of Django project organisation at the weekend. http://mauveweb.co.uk/posts/2014/08/organising-django-projects.html
May be of interest?
_______________________________________________ python-uk mailing listpython-uk@python.orghttps://mail.python.org/mailman/listinfo/python-uk
_______________________________________________ python-uk mailing listpython-uk@python.orghttps://mail.python.org/mailman/listinfo/python-uk
-- Jonathan Hartley tartley@tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
-- ------------------------------ Harry J.W. Percival ------------------------------ Twitter: @hjwp Mobile: +44 (0) 78877 02511 Skype: harry.percival
Old thread I know, but for the interested Brandon Rhodes' 'Clean Architecture' talk can be found at http://youtu.be/DJtef410XaM Steve
On 15 Aug 2014, at 11:22, Harry Percival <harry.percival@gmail.com> wrote:
Thanks Peter! I was speaking to Brandon at Pycon this year and he was telling me this was going to be his next talk to take on the road, and I was definitely looking forward to seeing it. Matt O'Donnell was also there, and he's done a talk on this sort of thing recently too (https://www.youtube.com/watch?v=NGhL7IA6Dik). It's definitely in the air.
My own modest attempts to approach the subject are in my book -- in chapter 19, where I show how striving for test isolation can (theoretically) push you towards something like a lean architecture (http://chimera.labs.oreilly.com/books/1234000000754/ch19.html) and in chapter 21, the wrap-up, where I waffle on about all these things (http://chimera.labs.oreilly.com/books/1234000000754/ch22.html)
I don't think I managed to broach the subject nearly as cleanly as Brandon did. I really admire his talks. His data structures talk was one of the top 3 I saw at Pycon this year (http://pyvideo.org/video/2571/all-your-ducks-in-a-row-data-structures-in-the...). Perfect pace, slides that complement rather than repeat the talk, fascinating and useful content...
Anyways, back to our onions - I guess the thing that's always bothered me a bit about the "clean architecture" is that my main project (pythonanywhere) is "all boundaries", to use Gary Bernhardt's terminology. Or, to put it differently, I don't think we really have much in the way of "business logic". We just turn Http requests into commands that go to processes. There's really not much in the way of "logic" in the way. No calculations or business rules to speak of. So it's never seemed worth it, to us.
And sometimes I think -- aren't many web projects just thin CRUD wrappers around a database? Is going to all the trouble of isolating your business logic from, eg, django, really worth it in most cases?
On 13 August 2014 13:09, Daniel Pope <lord.mauve@gmail.com> wrote: Coincidentally, I blogged on the topic of Django project organisation at the weekend.
http://mauveweb.co.uk/posts/2014/08/organising-django-projects.html
May be of interest?
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
-- ------------------------------ Harry J.W. Percival ------------------------------ Twitter: @hjwp Mobile: +44 (0) 78877 02511 Skype: harry.percival _______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
On Tue, 2012-12-04 at 14:46 +0000, Jonathan Hartley wrote:
I haven't, yet, but I'm thinking of refactoring a vertical slice of our monster Django app into this style, and I'd love to hear if you think it's crazy / brilliant / obvious / old-hat, etc.
All three, I suspect. Abstraction and properly designed APIs are always good, and I guess the layered structure occurs quite naturally. (or, perhaps, _should_ occur quite naturally?) My own difficulty with these structures comes from the business logic part. There's four aspects of this - the choice of what goes on the form presented to the user, the validation of user input, the logic of what happens when a button is pressed (calculations and storage), and the choice of what the user sees next (business process). The middle two can clearly be devolved to a business object, and the outer two can be _supported_ by a business object, but, it seems to me, there is always something 'businessy' in the user interface parts. My experience here is, admittedly, limited, so I'm probably missing something. I'm currently using a version of MVC, with components. I partition the source along business lines with appropriate parts (components) going under the headings of M, V, or C as appropriate. Creating a new business object or process can be a bit mind bending, but, generally, the bits fall into the places where you might expect them. None of this (for certain values of None) generates html directly, so, in principle, and within limits, I can customise what the user sees. So, I think I have separated concerns where this is possible, and not done so where it's just too difficult. Business logic is packaged in a way that makes it accessible, and, at the same time, recognises that the presentation is a factor. My conclusion, for what it's worth, is: layering is good, but don't be officious :-) Mike S.
participants (14)
-
Andy Robinson -
Chris Withers -
Daniel Pope -
Hansel Dunlop -
Harry Percival -
James Broadhead -
John Lee -
Jonathan Hartley -
Matt Hamilton -
Menno Smits -
Michael Foord -
Mike de Plume -
Peter Inglesby -
steve.allison