Mailman 3 March 2013 - Python-ideas

Message passing for objects
by Mark Janssen March 18, 2013

March 18, 2013

Just the thing I'm talking about. So I'm not the only one trying out these ideas..... :) (via the p2p-foundation mailing list) ---------- Forwarded message ---------- From: ProjectParadigm-ICT-Program <metadataportals(a)yahoo.com> Date: Sat, Mar 16, 2013 at 3:12 PM Subject: Re: A Distributed Economy -- A blog involving Linked Data To: Brent Shambaugh <brent.shambaugh(a)gmail.com>, Michel Bauwens <michel(a)p2pfoundation.net> Cc: Samuel Rose <samuel.rose(a)gmail.com>, "public-lod(a)w3.org" <public-lod(a)w3.org>, Paul Cockshott <william.cockshott(a)glasgow.ac.uk> Dear Brent, Thanks for bringing up this issue again, and let me rephrase. When considering resilience there are three networked systems to consider: humans, things and ideas/data/information. All three somehow now mesh and interact through the Internet and the Internet of Things. The conceptual model for the Semantic Web incorporates all three. Now if we want to complete this model we must incorporate the (Global) Ecosystems of Planet Earth. Thus we are able to create economic models that incorporate security, resilience, reliability and sustainability. As we speak thousands and thousands of engineers and scientists are already tackling the fundamental task of coming up with new all-encompassing paradigms and the 7th IEEE International Conference on Digital Ecosystems and Technologies will deal with quite a few of these issues. I will soon be setting up a blog which will deal with resilience, more specifically resilient grids and resilient grid technologies. The theme of resilience whether it relates to cyber space, natural ecosystems and resources management as part of sustainable economic development, human society or economic assets vis a vis natural disasters or conflict is proving to be be one of the hottest new themes in funded research, and one main central theme in Horizon 2020, the new European Union 71 billion euro research fund. Milton Ponson GSM: +297 747 8280 PO Box 1154, Oranjestad Aruba, Dutch Caribbean Project Paradigm: A structured approach to bringing the tools for sustainable development to all stakeholders worldwide by creating ICT tools for NGOs worldwide and: providing online access to web sites and repositories of data and information for sustainable development This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. ________________________________ From: Brent Shambaugh <brent.shambaugh(a)gmail.com> To: Michel Bauwens <michel(a)p2pfoundation.net> Cc: ProjectParadigm-ICT-Program <metadataportals(a)yahoo.com>; Samuel Rose <samuel.rose(a)gmail.com>; "public-lod(a)w3.org" <public-lod(a)w3.org>; Paul Cockshott <william.cockshott(a)glasgow.ac.uk> Sent: Thursday, March 14, 2013 7:12 PM Subject: Re: A Distributed Economy -- A blog involving Linked Data In an attempt to understand the conversation we had, I was sent in a flurry of confusion. I started checking out books, and one was Resilience by Andrew Zolli and Ann Marie Healy. I found a few quotes that seem exciting to me: "Adhocracies thrive on data. And by the stroke of fantastic luck, we're currently witnessing the global birth of an adhocracy of data -- a global revolution that, for the first time, empowers orgranizations with the capacity to collect and correlate widely distributed real-time information about the way many critical systems are performing. This kind of open data will play a central role in resilience strategies for years to come." pg. 266, Resilience, Andrew Zolli and Ann Marie Healy "And for organizations of all types there is a powerful lesson here: Resilience benefits accrue to organizations that prioritize the collection, presentation, and sharing of data." pg. 269, Resilience, Andrew Zolli and Ann Marie Healy "A related theme in the resilience discussion is the importance of networks, which provide a universal, abstract reference system for describing how information, resources, and behaviours flow GSM: +297 747 8280 PO Box 1154, Oranjestad Aruba, Dutch Caribbean Project Paradigm: A structured approach to bringing the tools for sustainable development to all stakeholders worldwide by creating ICT tools for NGOs worldwide and: providing online access to web sites and repositories of data and information for sustainable development This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. ________________________________ From: Brent Shambaugh <brent.shambaugh(a)gmail.com> To: Michel Bauwens <michel(a)p2pfoundation.net> Cc: ProjectParadigm-ICT-Program <metadataportals(a)yahoo.com>; Samuel Rose <samuel.rose(a)gmail.com>; "public-lod(a)w3.org" <public-lod(a)w3.org>; Paul Cockshott <william.cockshott(a)glasgow.ac.uk> Sent: Thursday, March 14, 2013 7:12 PM Subject: Re: A Distributed Economy -- A blog involving Linked Data In an attempt to understand the conversation we had, I was sent in a flurry of confusion. I started checking out books, and one was Resilience by Andrew Zolli and Ann Marie Healy. I found a few quotes that seem exciting to me: "Adhocracies thrive on data. And by the stroke of fantastic luck, we're currently witnessing the global birth of an adhocracy of data -- a global revolution that, for the first time, empowers orgranizations with the capacity to collect and correlate widely distributed real-time information about the way many critical systems are performing. This kind of open data will play a central role in resilience strategies for years to come." pg. 266, Resilience, Andrew Zolli and Ann Marie Healy "And for organizations of all types there is a powerful lesson here: Resilience benefits accrue to organizations that prioritize the collection, presentation, and sharing of data." pg. 269, Resilience, Andrew Zolli and Ann Marie Healy "A related theme in the resilience discussion is the importance of networks, which provide a universal, abstract reference system for describing how information, resources, and behaviours flow through many complex systems. Having a common means to describe biological, economic, and ecological systems, for example, allows researchers to make comparisons between the ways these very different kinds of entities approach similar problems, such as stopping a contagion - whether an actual virus, a financial panic, an unwanted behavior, or an environmental contaminant - when it begins to spread. Having a shared frame of reference allows us to consider how successful tactics in one domain might be applied to another - as we'll see in newly emerging fields like ecological finance." pg 19, Resilience, Andrew Zolli and Ann Marie Healy "Rather the resilience frame suggests a different, complementary effort to mitigation: to redesign our institutions, embolden our communities, encourage innovation and experimentation, and support our people in ways that will help them be prepared and cope with surprises and distruptions, even as we work to fend them off." pg 23, Resilience, Andrew Zolli and Ann Marie Healy It is interesting that Buckminster Fuller wrote about similar ideas over 30 years ago: "The inefficiency of automobiles' reciprocating engines - and their traffic-system-wasted fuel - and the energy inefficiency of today's buildings, are only two of hundreds of thousands of instances that can be cited of the design-avoidable energy wastage. But the technical raison d'etre for either the energy-effectiveness gains or losses is all completely invisible to human eyes. Thus, the significance of their omni-integratable potentialities is uncomprehended by either the world's leaders or the led. Neither the great political and financial power structures of the world, nor the specialization-blinded professionals, nor the population in general realize the sum-totally the omni-engineering-integratable, invisible revolution in metallurgical, chemical, and electronic arts now makes it possible to do so much more with ever fewer pounds and volumes of material, ergs of energy, and seconds of time per given technological function that it is now highly feasible to take care of everybody on Earth at a "higher standard of living than any have ever known.", pg. xxv, Critical Path, R. Buckminster Fuller "World Game will become increasingly effective in its prognoses and programming when the world-around, satellite-interrelayed computer system and its omni-Universe-operative (time-energy) accounting system are established. This system will identify the kilowatt-hour-expressed world inventory of foods, raw and recirculating resources, and all the world's unique mechanical and structural capabilities and their operating capacities as well as the respective kilowatt-hours of available energy-income-derived operating power with which to put their facilities to work. All the foregoing information will become available in respect to all the world-around technology's environment-controlling, life-sustaining, travel- and communication-accomidating structures and machines.", pg. 219, Critical Path, R. Buckminster Fuller I'm happy that Milton Ponson pointed out Resilience. I had never thought about resilience before. Looking into it was very gratifying. It gave me some confidence that I was perhaps doing some things right, but at the same time startled me by how much there is to learn to somehow survive the free fall. Doing a search for Linked Data and Resilience gave me a result from rkbexpolorer (http://www.rkbexplorer.com/explorer/#display=project-{http%3A//wiki.rkbexpl…) which is from the ReSIST (Resilience for Survivability) project in Europe (http://www.resist-noe.org/). They also have some links to some free course material at <http://resist.isti.cnr.it/home.php>. I believe my blog evolved to explore a peer-to-peer economy. Michael Bauwens desribes such economies as distributed networks, "As political, economic, and social systems transform themselves into distributed networks, a new human dynamic is emerging: peer to peer (P2P). As P2P gives rise to the emergence of a third mode of production, a third mode of governence, and a third mode of property, it is poised to overhaul our political economy in unprecendented ways." (www.ctheory.net/articles.aspx?id=499) This suggests something broader. As a result of our conversation, I also looked at some people with socialist views such as Roberto Verzola, W. Paul Cockshott and Allin Cottrell, Raoul Viktor, and Heinz Dieterich. Roberto Verzola describes an economy of abundance, which may indeed be linked to P2P technologies. "An economy of abundance seeks to dismantle or reform these scarcity-generating institutions in such a way as to affirm our freedom to live life as art (self-expression to others), social equity (so that everything can live life as art), and sustainability (so that all life can thrive into the future). Among other things, this implies a much greater role for various forms of shared property, individual an community-level self-reliance, and participatory decision making." (http://www.shareable.net/blog/event-the-economics-of-abundance) He also argues that for innovation to proceed, everyone seeking knowledge should have access to it. "the most important means to ensure that innovation can proceed is to ensure that everyone seeking knowledge has access to it. ... Knowledge that helps empower people depends on openness, while knowledge that is used to coerce, to exert power over the disempowered, thrives on secrecy" p. 150, The Economics of Abunance: A Political Economy of Freedom, Equity and Sustainability, Roberto Verzola This seems to align well with my present feelings. I feel that engineering is so saturated with IP, that it is hard to feel like you're not going to be doing that. At the same time I want to develop my skills and thrive. How do become a Professional Engineer and not feel like you're going to be doing that? What if you don't like the lawyer saturated culture where people are suing other people over some idea you or someone else produced? I can sense that a lot of people, especially in the hacker and maker community, want to be able to support themselves and work on cool new things but don't want to deny other people to work on the same cool things. Why do ideas have to take on a life of their own and become part of something you might be employed by, but have no control beyond that? Sorry about the extreme language, but why do I imagine it as making deal with the devil? Betraying your friends so you can enjoy life and eat? In this, there is an underlying assumption that there are institutions that do not want to share or partner, or make it very difficult. If it is easier, I feel that could be better. Buckminster Fuller also wrote about such things: "2. Grandmother taught us the Golden Rule: "Love thy neighbor as thy self--do unto others as you would they should do unto you. 3. As we became older and more experienced, out uncles began to caution us to get over our sensitivity. "Life is hard," they explained. 'There is nowhere nearly enough life support for everybody on our planet, let alone enough for a comfortable life support. If you want to raise a family and have a comfortable life for them, you are going to have to deprive many others of the opportunity to survive and the sooner, the better. Your grandmother's Golden Rule is beautiful, but it doesn't work.'" p. 123. Critical Path, R. Buckminster Fuller Is it possible to have a win-win between people an business? Are there any financial barriers to entry and/or partnership? Sometimes I fear that I will never be paid enough to implement my ideas, or if I do then it will be too late to enter the market. Either I can't afford to do the work, or someone who developed and patented something that matches at least some of my idea decides not to involve me. Thus I would question spending the time developing my idea. I'm assuming I can develop my idea for my own personal use (possibly not?). However, I am more certain I may have trouble sharing and selling things developed from my ideas. I imagine that this favors those who already have money. People like Eric von Hippel and Michael Bauwens both speak about a lot of innovation goinf on outside the firm. For example Michel Bauwen's states: "The French-Italian school of 'cognitive capitalism' stresses the value creation today is no longer confined to the enterprise, but beholden to the mass intellectuality of knowledge workers, who through their lifelong learning/experiencing and systematic connectivity, constantly innovate within and without the enterprise. This is an important argument, since it would justify what we see as the only solution for the expansion of the P2P sphere into a society at large: the universal basic income. Only the independence of work and the salary structure can guarantee the peer producers can continue to create this sphere of highly productive use value." The Political Economy of Peer Production (www.ctheory.net/articles.aspx?id=499) Eric von Hippel also speaks about his book. (http://commons.wikimedia.org/wiki/File:20060123-Eric.von.Hippel-Democratizi…) However, giving people money for doing nothing makes me feel uncomfortable. Of course, as Heinz Dietrich suggests, if you already have money, things work just fine: "The first step, in fact, would be to establish a new cybernetic principle; you need something that coordinates billions of economic transactions everyday. And, so far, the market has been a relatively well-functioning system under two conditions: If the market is not monopolistic and you have the buying power for the merchandise you produce and for the services, then the market coordinates quite well."-- The Socialism of the 21st Century (http://eipcp.net/transversal/0805/dieterich/en) Paul Cockshott, and Allin Cottrell suggest a payment system determined democratically, "The payment system outlined in chapter 2 depends on the idea that the total labour content of each product or service can be calculated." p. 8, Towards a New Socialism If this is by the state, then I am moved to say I do not trust the government to do much right at all. Certainly, this is what I feel if I spend any length of time watching the news. But I would like to look into it. If this develops into something, the state should be involved at some level. I feel bad about this. I'll have to read more. However, I agree that with more democracy things would be better. "The principal bases for a post-Soviet socialism must be radical democracy and efficient planning. The democratic element, it is now clear, is not a luxury, or something that can be postponed until conditions are especially favourable. Without democracy, as we have argued above, the leaders of a socialist society will be driven to coercion in order to ensure the production of a surplus product, and if coercion slackens the system will tend to stagnate." p. 7, Towards a New Socialism I definately think there needs to be some way to accomplish things that makes it fair to people. In terms of me, I believe this underlies a larger problem than me being connected with the right job or being afraid of debt going back to school. It is the problem of connecting people with the right jobs, utilizing the skills they already have so they don't have to fear paying to learn what they already know, and raising awareness that the jobs are there. I dream about linked data being able to illuminate relationships between present skills and related skills to job seekers and employers. I also dream about linked data allowing people to market themselves with clarity as a basket of skills that represents who they really are rather than a basket of skills that was set by a well-meaning college, trade-school, or university. I honestly believe that people who do something they have a passion for, will be more effective employees or entrepreneurs. But how to pay for it? If you take out debt you need to find a way to pay it off. If you can't find something that reflects your values you may feel like you're enslaved to something else while trying to pay it off. It can be a pressing struggle as Paul Grignon's Money as Debt video on Youtube describes (https://www.youtube.com/watch?v=0K5_JE_gOys). Paul Grigon and others say that our present monetary system leads to infinite growth. "We need to become politically sensitive to the invisible architectures of power. In distributed systems, where there is no overt hierarchy, power is a function of design. One such system, perhaps the most important of all, is the monetary system, whose interest-bearing design requires the market to be linked to a system of infinite growth, and this link needs to be broken. A global reform of the monetary system, or the spread of new means of direct social production of money, are the necessary conditions for such a break." (http://p2pfoundation.net/About_The_Foundation) I'd imagine this would create no problems for people as long as there is the will and resources to grow infinitely. However, Paul Grigon points out an exception: those with the money to lend at interest will eventually wind up with all of the money, and due to forclosure the property too. My site explores distributed funding. (http://adistributedeconomy.blogspot.com/2012/03/distributed-funding.html). I am still not sure how exactly to accomplish it. I think it may involve something like Ripple (https://classic.ripplepay.com/) and PaySwarm (http://payswarm.com/). A friend of mine pointed out that it did not seem that Ripple allows to keep track of what you owe who, whereas PaySwarm appears to do so. I may need to develop something on my own (http://lists.w3.org/Archives/Public/public-webpayments/2013Feb/0034.html) that involes donations, and whatever models are needed. Embarrasingly, I'm still learning JavaScript. Thankfully, my friends are also encouraging me to focus on some small project. A few other thoughts: As I was reading, I noticed some mention of rival and non-rival goods. Rival goods could be seen as raw materials or products, and non-rival goods could exist in an infinite amount. In the maker world I see things such as CAD files as non-rival and raw materials and end products as rival. I question whether people would still pay for rival goods, and perhaps donate for non-rival goods if there was an open source economy? What if things such as PaySwarm made it easy to do so? The Rep Rap and Ardrino are open source hardware, and all products by Makerbot used to be (http://josefprusa.cz/open-hardware-meaning/, http://www.hoektronics.com/2012/09/21/makerbot-and-open-source-a-founder-pe…). People can share their designs, but would people share their profits with those who contributed to their idea? It wouldn't have to be much, as small amounts still add up. Would this be bad? Even if people don't have to pay, things might still vary as does the amount you might get by selling a book? Concieveably if you have a lot of open source hardware, then you could have as much flexibility in the physical world as you do in the software world. In an extreme case, maybe you could have open source spaceships. They are after all lots of little parts, much like a GNU/Linux distro. If things could be freely copied and not exclusively owned as in the GPL, would you still have brand loyalty? While not going into the fine details, the Ultimaker and the Makerbot Thing-O-Matic look very similar. Why would I want one over the other? If whatever you chose was linked to previous innovations, and people let their donations flow to those authors, how much would it matter? Would the crowd maintain accountability so people would not collect money for doing nothing? The maker community seems to be supportive of things that they are free to contribute to. How far could this go, especially with support from arguments made by Don Tapscott and Anthony D. Williams in the book Wikinomics? For the legality, things like the JOBs act seem exciting. However, this seems to be for equity-based crowdfunding and not just donations (https://en.wikipedia.org/wiki/JOBS_Act). I'd imagine that it would be both even if some of the things were as described above. What if you were retired, and you had money, but nothing you contributed was being used? Could you grow your money to support yourself? The potential of digital technologies seems huge. I read about the Industrial Internet, as pointed out by Milton Poson. The GE report titled, "Industrial Internet. Pushing the Boundaries of Minds and Machines"<www.ge.com/docs/chapters/Industrial_Internet.pdf>, by Peter C. Evans and Marco Annunziata has the following quote: "The combination of physics- based approaches, deep sector specific domain expertise, more automation of information flows, and predictive capabilities can join with the existing suite of “big data” tools. The result is the Industrial Internet encompasses traditional approaches with newer hybrid approaches that can leverage the power of both historic and real-time data with industry specific advanced analytics." Of course, this makes me want to go down the path of Density Functional Theory and Molecular Dynamics. I had a brief exposure to these concepts in graduate school, and it reminds me of the layout algorithms in Gephi (at least MD, I know a little less about DFT). Yes! I just learned about DBSCAN (http://en.wikipedia.org/wiki/DBSCAN) in R (http://www.cl.cam.ac.uk/~dq209/others/Rdatamining.pdf) and ELKI (http://en.wikipedia.org/wiki/ELKI). And here you have it, the original DBSCAN paper (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.121.9220). In addition, they speak about Enterprise Management Software in terms of the Industrial Internet: "At the other end of the spectrum, enterprise management software and solutions have been widely adopted to drive organizational efficiencies at the firm level. The benefits of these efforts include better tracking and coordination of labor, supply chain, quality, compliance, and sales and distribution across broad geographies and product lines. However, these efforts have sometimes fallen short because while they can passively track asset operations at the product level, the ability to impact asset performance is limited. Optimizing the system to maximize asset and enterprise performance is what the Industrial Internet offers." This reminds me of a presentation given by Dr. Manoj Dharwadkar of Bentley Systems Inc. titled, "Using Sematic Web Technologies in Open Applications" (http://www.w3.org/2008/12/ogws-slides/Bentley.pdf). It also reminds me of the The Simantics Platform at the VTT Technical Research Centre of Finland (www.vtt.fi) and the mission of Dassault Systems (www.3ds.com). They all have the ISO15926 ontology in common. I wasn't sure if they were talking about linked data in the report: "To make information intelligent, new connections need to be developed so that Big Data ‘knows’ when and where it needs to go, and how to get there. If imaging data is better connected, the right doctor could automatically receive a patient’s rendered images so the information is finding the doctor instead of the doctor finding the information. " --- Opportunity for Liked Data? random paper with medical devices communicating with the semantic web: http://radiographics.rsna.org/content/30/7/2039.abstract http://www.mdpnp.org/uploads/8-3_Schluter_26Jan.pdf (devices commmunicating, like Industrial Internet) Further, they go into what is needed to to build the Industrial Internet: "The Industrial Internet will require an adequate backbone. Data centers, broadband spectrum, and fiber networks are all components of the ICT infrastructure that will need to be further developed to connect the various machines, systems, and networks across industries and geographies. This will require a combination of inter- and intra- state infrastructure order to support the significant growth in data flows involved with the Industrial Internet. " I heard that Oklahoma, and the U.S. in general, needs more fiber. Someone said that Denver, Dallas, Kansas City, Silicon Valley, Austin(?), all have good networks. How would the talent to build the Industrial Internet be gathered? Here are a few more quotes: "Other alternatives for sourcing cross-discipline talent might include developing the existing resources in the native domain through collaborative approaches. Instead of building or buying talent that has multiple skills, create environments that accelerate the ability of people with different skills to interact and innovate together. On a larger scale, approaches such as crowdsourcing might be able to close some of the capabilities gaps that are sure to occur." "Today, the people that manage big data systems or perform advanced analytics have developed unique talents through self-driven specialization, rather than through any programs that build a standard set of skills or principles. Co-development of curriculum, integration of academic staff into industry, and other approaches will be needed to ensure that the talent needs of the Industrial Internet do not outpace the educational system." There definately is a lag between the development of IT, and its adoption. In Chemical Engineering, I'm pretty sure people thought I was crazy when I started talking about the Semantic Web. People in network security, and even computer science were not familiar with it. If you're talking about Wikinomics (Openness, Peering, Sharing, Acting Global) thinking there might be some growth to do. I heard of people at universities and hackerspaces speak of themselves as universities, but their culture is very different. Maybe hackerspaces are on the extreme of being open, whereas universities are less so? Maybe this is the case with IP. Maybe less so, with papers (but who can access them?). See a presentation in 2008 by RIC Jackson, then Director of the FIATECH Consortium: (http://www.slccc.net/documents_pdf/Technology-Ric%20Jackson%200811.pdf). Adoption of new tech for the enterprise is slow: (http://pandodaily.com/2012/02/11/why-oracle-may-really-be-doomed-this-time/). There are some, such as the Mayor of Newark, NJ, who bothered to go to SXSW to speak about the adoption of more tech: (http://pandodaily.com/2013/03/10/cory-booker-calls-for-tech-empowered-open-…). Here are three more quotes from the report: "Measures to ensure the security of restricted data, including intellectual property,proprietary information, and personally identifiable information (PII) are critical. " --- this reminds me of the Read Write Web Community Group "Currently there are several standards bodies, but they are fragmented. The promotion and adoption of common and consistent standards on data structure, encryption, transfer mechanisms, and the proper use of data will go a long way in advancing cyber security." I was made fun of by a CS graduate when I was excited about a possible new standard. "Academia: Further research on data security and privacy should be pursued, including research on enhancing IT security, metrology, inferencing concerns with non- sensitive data, and legal foundations for privacy in data aggregation." Perhaps more collaboration with the hacker community? Is it true that some programmers, and some in CS tend to build things and ignore security in the process? I wonder what is going on at hacker conferences like Blackhat and DEFCON. BTW, people at the Chaos Communication Congress are geniuses. Whew! That's enough. If you're interested in more, read the report. It's exciting. :) I asked myself this question: What is the future role of the University? The university may serve as a repository for books, a place to do research, and a meeting place. Lectures? I'm not sure. How do the things that Michael Hammer & Lisa W. Hershman talk about fit in? They wrote a book titled, "Faster, Cheaper, Better: The 9 Levers for Transforming How Work Gets Done". I believe they were talking about Business Process Improvement (https://en.wikipedia.org/wiki/Business_process_improvement). Would Process Owners, as mentioned by them, serve a major role in the Industrial Internet? (http://it.toolbox.com/wiki/index.php/Process_Owner) ---------------- Resources that I am considering reading: The Net Delusion: The Dark Side of Internet Freedom, Evengy Morozov, 2011 To Save Everything, Click Here: The Folly of Technological Solutionism, Evengy Morozov, 2013 The Wealth of Networks, Yochai Benkler, Yale University Press, 2006 Science and the Crisis in Society, Frank H. George, Wiley, 1970 Future Perfect: The Case for Progress in a Networked Age, Steven Johnson, 2012 Here Comes Everybody: The Power of Organization Without Organizations, Clay Shirky, 2008 Nasa's Advanced Automation for Space Missions, http://www.islandone.org/MMSG/aasm/ (Robots, Expert Systems, Etc..), The Technical Stuff, 1980 The Future: Six Drivers of Global Change, Al Gore, 2013, http://www.amazon.com/The-Future-Drivers-Global-Change/dp/0812992946, Reviewed by Tim Berners-Lee, may relate well to the previous link An Inquiry to Into the Nature and Causes of the Wealth of Nations, Adam Smith, LL.D. F.R.S., MDCCCXLIII (1843) (according to Wikipedia, it was first published in 1776) Books by Chris Anderson and Lawrence Lessig Consent of the Networked: The Worldwide Struggle for Internet Freedom, Rebecca MacKinnon, 2012 Business Process Improvement: the breakthrough strategy for total quality, productivity, and competitiveness, H J Harrington , 1991 Faster, better, cheaper : low-cost innovation in the U.S. space program, Howard E McCurdy, 2001 On Wed, Jan 9, 2013 at 11:42 PM, Michel Bauwens <michel(a)p2pfoundation.net> wrote: it seems to me that these shifts have already started, before 2013, including in these fields, but are also much more long-term transformations ... in the case of deep-pocketed and politically powerful vested interests, only moderate bottom-up advances can be expected in the very short term ... both telecom and banking are still heavily centralized, they enabled people-based p2p dynamics but control the infrastructure, the data, the design and many other aspects of their only partly distributed systems ... I'm sure the same is true of GE .. no corporation will allow a fully p2p distributed system without some form of centralized control Michel On Thu, Jan 10, 2013 at 2:54 AM, ProjectParadigm-ICT-Program <metadataportals(a)yahoo.com> wrote: 2013 will see the advent of new paradigms for infrastructures that up until now where centralized, i.e. electric power generation and distribution, intermodal transportation and logistics, food and agro-industrial production and distribution, industrial production and distribution, consumer products manufacturing and distribution, pharmaceuticals production and distribution, energy extraction and distribution (including coal, gas, shale oil/gas and biofuels). The data and telecom infrastructure and parallel the banking and financial sectors are the only ones espousing decentralized distributed P2P (and B2B) processes. Resilience is a property that can only be achieved by copying the structure of the internet and some of its inherent characteristics. By defining strategic infrastructures as decentralized networks of distributed P2P (B2B) processes embedded in an intelligent grid it becomes possible to define resilience in a way similar to the resilience of the Internet. And a resilient grid lends itself perfectly to embedding in a semantic web overlay grid. The Industrial Internet as defined by GE and outlined in a recent white paper comes pretty close to it but not quite yet. See http://www.gereports.com/meeting-of-minds-and-machines/. Milton Ponson GSM: +297 747 8280 PO Box 1154, Oranjestad Aruba, Dutch Caribbean Project Paradigm: A structured approach to bringing the tools for sustainable development to all stakeholders worldwide by creating ICT tools for NGOs worldwide and: providing online access to web sites and repositories of data and information for sustainable development This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. ________________________________ From: Brent Shambaugh <brent.shambaugh(a)gmail.com> To: ProjectParadigm-ICT-Program <metadataportals(a)yahoo.com> Cc: Michel Bauwens <michel(a)p2pfoundation.net>; Samuel Rose <samuel.rose(a)gmail.com>; "public-lod(a)w3.org" <public-lod(a)w3.org>; Paul Cockshott <william.cockshott(a)glasgow.ac.uk> Sent: Wednesday, January 9, 2013 2:10 AM Subject: Re: A Distributed Economy -- A blog involving Linked Data Oh well, I'll share my story on a W3C forum no less. Model, true. Would my experiences even translate? I think you'd have to see this from my own personal perspective. Even though I grew up in an American home there was a lot of discourse in my family. There wasn't a lot of room for personal expression, and my family was very religious so I was afraid of offending God if I went against the dominating figure and/or ideology in the family. I was also very shy growing up, and I did not have much money, even though I came from an upper middle class family. I felt out of place most of the time, and sometimes I had ideas that people did not seem to understand or be interested in. I liked computers, and wanted to learn more about them. I was always asking people doing computer stuff how to program, even though I had a lot of trouble doing it myself. I think it was because I struggled with algebra (and other maths), but more so algebra. I also was a bit lost in some documentation, and may have not been fully aware of other resources that may have helped. I was afraid of tinkering, but I built webpages and was proud of them and I also built structures in the woods (but that is a bit off topic). My family paid for my college. I'm thankful for that, but it also leaves me with a feeling of responsibility to them. I'll admit to not being in sync with things in my undergraduate years. It looks very good if you have an internship. But at the time I made a few mistakes perhaps. I was a bit afraid to try because the companies I qualified for either were not doing something that interested me and/or something that I felt reflected my beliefs, values and possibly something else that is hard to describe. In short, perhaps passion. Over time I realized that it would probably be wiser to accept things as is if I ever hoped to be employed. Making the sale was difficult though. I think perhaps people think I'm lazy, or uniformed, because I did not work (except for academic things) in college. Or was it emotion? Ideas out of place? I was also affected by many of the same family things growing up. I have an interest in physics, electronics, economics, systems, etc. I think that if I ever hope to use my education, and share what I have learned, I need to do something amazing. I could go back to school, take on a lot of debt, and just hope that I get enough good grades to impress enough people (and not have them think I'd get bored when trying to get a job). Or I could learn things on my own, participate in projects, and hope that people receive me with open arms. Since 2007 when I discovered Polywell nuclear fusion I've gained new perspective on the world. I never actually built a fusion reactor, but I did try to learn what was behind them. This motivated me to read lots of books, and my desire to do other things to explore my uniqueness as an individual led to even more books. GNU/Linux facilitated my graduate work, and I can relate to it's philosophy through my many frustrations. Open source is great, because I don't have to worry so much about my skills wasting away. Being at the university also helps. I also don't have to manufacture things or do anything special to have excitement about it. But you know, how much can you actually get from someone who hasn't experienced that much real employment? Because of that automatically people see me in a certain way. And my views may not be necessarily realistic for lack of experience. But whatever it is, it seems I have have found a lot of energy and my friends seem to notice. I think about what I am learning more too. But would this model help people in the real world? I feel that had it existed it could have helped me growing up, but that is my own personal experience. In addition to studying, a lot of my peers spent their time drinking beer, socializing, and playing and/or watching sports. And most seemed to have more money. Now most seem to have even more money, and spend time on Facebook talking about things they have bought or families that they are raising. Their educational level is hard to discern. Not many seem to be posting things about hacking, making or things that might suggest deep insight. But not everyone fits that. I guess what matters is whether it will work or not, and whether it truly will benefit others. For that both an experiment and conversation will help. Thank you Samuel for referring me to Michael. Milton, I am not certain what it will do yet. I am not certain what resilience truly means. I'm definitely bothered by the wastefulness brought upon by obsolescence of products. It would be much better I think if we knew how they worked so we could reuse the them (I'm saying the parts) in other things. We've had this problem at the hackerspace. We have lots of stuff around that if we had the blueprint, it would be much better. If we knew how this blueprint connected to other things I personally think that would be even better. On a separate issue. In graduate school there were people there that seemed really lost. I mean they were doing their work, but didn't seem to have a joy about it. There also was not a lot of organization, and it was hard to find things. Outside of school, there are people that I know could go to graduate school but didn't. It was frustrating to me that I could not seem to sell them on thinking more deeply about things, or when they said I was really smart (but did not have the confidence or belief that they could do it themselves). Still others just weren't there. I've seen those who weren't there at the hackerspace. I question why, and think the world would be a better place if this could be tapped into. " Roberto Verzola is to my mind the political economist who has done most in studying this, see http://p2pfoundation.net/Category:Commons_Economics ; Wolfgang Hoeschele is planning an ambitious database based on a Needs, Organisational REsources, (I forgot what the A stands for) I'm sure that the proposed modelling effort will contribute to this field; if you are ideologically open, you may also want to talk with people like Paul Cockshott and the people of the Center for Transition Science at UNAM in Mexico City; who are very good at econometric modelling and interested in a cybernetic planning revival, " I still have to think more about this. I was reading over it a bit today. I might have seen something about this today. Someone was talking about how technologies were allowing us (or could? ) to become more mobile, and that people really didn't have to be co located. I don't remember what technologies that they were referring. "Peer to peer processes in addition should be defined as geography independent, historically nomads, hunter gatherers and technomads in the modern age all show this to be true." I hope to write soon. On Tue, Jan 8, 2013 at 6:57 PM, Brent Shambaugh <brent.shambaugh(a)gmail.com> wrote: I'm feeling that this is shaped by my own personal experience? I'm willing, but should I risk putting it out there? -- P2P Foundation: http://p2pfoundation.net - http://blog.p2pfoundation.net Updates: http://twitter.com/mbauwens; http://www.facebook.com/mbauwens #82 on the (En)Rich list: http://enrichlist.org/the-complete-list/ -- P2P Foundation: http://p2pfoundation.net - http://blog.p2pfoundation.net Updates: http://twitter.com/mbauwens; http://www.facebook.com/mbauwens #82 on the (En)Rich list: http://enrichlist.org/the-complete-list/ _______________________________________________ P2P Foundation - Mailing list http://www.p2pfoundation.net https://lists.ourproject.org/cgi-bin/mailman/listinfo/p2p-foundation

2 1

new PEP submission: transitioning to hosting release files on PYPI
by holger krekel March 16, 2013

March 16, 2013

Hi all, FYI i just submitted the below PEP draft to the PEP editors to receive a proper version number. The topics and the draft saw extensive discussions (four versions) on catalog-sig(a)python.org. We'd like to keep the discussion there if possible. best, holger krekel PEP: XXX Title: Transitioning to release-file hosting on PyPI Version: $Revision$ Last-Modified: $Date$ Author: Holger Krekel <holger(a)merlinux.eu>, Carl Meyer <carl(a)oddbird.net> Discussions-To: catalog-sig(a)python.org Status: Draft (PRE-submit V4) Type: Process Content-Type: text/x-rst Created: 10-Mar-2013 Post-History: Abstract ======== This PEP proposes a backward-compatible two-phase transition process to speed up, simplify and robustify installing from the pypi.python.org (PyPI) package index. To ease the transition and minimize client-side friction, **no changes to distutils or existing installation tools are required in order to benefit from the first transition phase, which will result in faster, more reliable installs for most existing packages**. The first transition phase implements an easy and explicit means for a package maintainer to control which release file links are served to present-day installation tools. The first phase also includes the implementation of analysis tools for present-day packages, to support communication with package maintainers and the automated setting of default modes for controlling release file links. The first phase also will default newly-registered projects on PyPI to only serve links to release files which were uploaded to PyPI. The second transition phase concerns end-user installation tools, which shall default to only install release files that are hosted on PyPI and tell the user if external release files exist, offering a choice to automatically use those external files. Rationale ========= .. _history: History and motivations for external hosting -------------------------------------------- When PyPI went online, it offered release registration but had no facility to host release files itself. When hosting was added, no automated downloading tool existed yet. When Philip Eby implemented automated downloading (through setuptools), he made the choice to allow people to use download hosts of their choice. The finding of externally-hosted packages was implemented as follows: #. The PyPI ``simple/`` index for a package contains all links found by scraping them from that package's long_description metadata for any release. Links in the "Download-URL" and "Home-page" metadata fields are given ``rel=download`` and ``rel=homepage`` attributes, respectively. #. Any of these links whose target is a file whose name appears to be in the form of an installable source or binary distribution, with name in the form "packagename-version.ARCHIVEEXT", is considered a potential installation candidate by installation tools. #. Similarly, any links suffixed with an "#egg=packagename-version" fragment are considered an installation candidate. #. Additionally, the ``rel=homepage`` and ``rel=download`` links are crawled by installation tools and, if HTML, are themselves scraped for release-file links in the above formats. Today, most packages released on PyPI host their release files on PyPI, but a small percentage (XXX need updated data) rely on external hosting. There are many reasons [2]_ why people have chosen external hosting. To cite just a few: - release processes and scripts have been developed already and upload to external sites - it takes too long to upload large files from some places in the world - export restrictions e.g. for crypto-related software - company policies which require offering open source packages through own sites - problems with integrating uploading to PyPI into one's release process (because of release policies) - desiring download statistics different from those maintained by PyPI - perceived bad reliability of PyPI - not aware that PyPI offers file-hosting Irrespective of the present-day validity of these reasons, there clearly is a history why people choose to host files externally and it even was for some time the only way you could do things. This PEP takes the position that there are at least some valid reasons for external hosting. Problem ------- **Today, python package installers (pip, easy_install, buildout, and others) often need to query many non-PyPI URLs even if there are no externally hosted files**. Apart from querying pypi.python.org's simple index pages, also all homepages and download pages ever specified with any release of a package are crawled by an installer. The need for installers to crawl external sites slows down installation and makes for a brittle and unreliable installation process. Those sites and packages also don't take part in the :pep:`381` mirroring infrastructure, further decreasing reliability and speed of automated installation processes around the world. Most packages are hosted directly on pypi.python.org [1]_. Even for these packages, installers still crawl their homepage and download-url, if specified. Many package uploaders are not aware that specifying the "homepage" or "download-url" in their package metadata will needlessly slow down the installation process for all users. Relying on third party sites also opens up more attack vectors for injecting malicious packages into sites using automated installs. A simple attack might just involve getting hold of an old now-unused homepage domain and placing malicious packages there. Moreover, performing a Man-in-The-Middle (MITM) attack between an installation site and any of the download sites can inject malicious packages on the installation site. As many homepages and download locations are using HTTP and not HTTPS, such attacks are not hard to launch. Such MITM attacks can easily happen even for packages which never intended to host files externally as their homepages are contacted by installers anyway. There is currently no way for package maintainers to avoid external-link crawling, other than removing all homepage/download url metadata for all historic releases. While a script [3]_ has been written to perform this action, it is not a good general solution because it removes useful metadata from PyPI releases. Even if the sites referenced by "Homepage" and "Download-URL" links were not scraped for further links, there is no obvious way under the current system for a package owner to link to an installable file from a long_description metadata field (which is shown as package documentation on ``/pypi/PKG``) without installation tools automatically considering that file a candidate for installation. Conversely, there is no way to explicitly register multiple external release files without putting them in metadata fields. Goals ----- These are the goals to be achieved by implementation of this PEP: * Package owners should be able to explicitly control which files are presented by PyPI to installer tools as installation candidates. Installation should not be slowed and made less reliable by extensive and unnecessary crawling of links that package owners did not explicitly nominate as installation files. * It should remain possible for package owners to choose to host their release files on their own hosting, external to PyPI. It should be easy for a user to request the installation of such releases using automated installer tools. * Automated installer tools should not install externally-hosted packages **by default**, but only when explicitly authorized to do so by the user. When tools refuse to install such a package by default, they should tell the user exactly which external link(s) they would need to follow, and what option(s) the user can provide to authorize the tool to follow those links. PyPI should provide all necessary metadata for installer tools to implement this easily and within a single request/reply interaction. * Migration from the status quo to the above points should be gradual and minimize breakage. This includes tooling that makes it easy for package owners with an existing release process that uploads to non-PyPI hosting to also upload those release files to PyPI. Solution / two transition phases ================================ The first transition phase introduces a "hosting-mode" field for each project on PyPI, allowing package owners explicit control of which release file links are served to present-day installation tools in the machine-readable ``simple/`` index. The first transition will, after successful hosting-mode manipulations by individual early-adopters, set a default hosting mode for existing packages, based on automated analysis. **Maintainers will be notified one month ahead of any such automated change**. At completion of the first transition phase, **all present-day existing release and installation processes and tools are expected to continue working**. Any remaining errors or problems are expected to only relate to installation of individual packages and can be easily corrected by package maintainers or PyPI admins if maintainers are not reachable. Also in the first phase, each link served in the ``simple/`` index will be explicitly marked as ``rel="internal"`` (hosted by the index itself) or ``rel="external"`` (linking to an external site that is not part of the index). In the second transition phase, PyPI client installation tools shall be updated to default to only install ``rel="internal"`` packages unless a user specifies option(s) to permit installing from external links. Maintainers of packages which currently host release files on non-PyPI sites shall receive instructions and tools to ease "re-hosting" of their historic and future package release files. This re-hosting tool MUST be available before automated hosting-mode changes are announced to package maintainers. Implementation ============== Hosting modes ------------- The foundation of the first transition phase is the introduction of three "modes" of PyPI hosting for a package, affecting which links are generated for the ``simple/`` index. These modes are implemented without requiring changes to installation tools via changes to the algorithm for generating the machine-readable ``simple/`` index. The modes are: - ``pypi-scrape-crawl``: no change from the current situation of generating machine-readable links for installation tools, as outlined in the history_. - ``pypi-scrape``: for a package in this mode, links to be added to the ``simple/`` index are still scraped from package metadata. However, the "Home-page" and "Download-url" links are given ``rel=ext-homepage`` and ``rel=ext-download`` attributes instead of ``rel=homepage`` and ``rel=download``. The effect of this (with no change in installation tools necessary) is that these links will not be followed and scraped for further candidate links by present-day installation tools: only installable files directly hosted from PyPI or linked directly from PyPI metadata will be considered for installation. Installation tools MAY evolve to offer an option to use the new rel-attribution to crawl external pages but MUST NOT default to it. - ``pypi-explicit``: for a package in this mode, only links to release files uploaded to PyPI, and external links to release files explicitly nominated by the package owner (via a new interface exposed by PyPI) will be added to the ``simple/`` index. Thus the hope is that eventually all projects on PyPI can be migrated to the ``pypi-explicit`` mode, while preserving the ability to install release files hosted externally via installer tools. Deprecation of hosting modes to eventually only allow the ``pypi-explicit`` mode is NOT REGULATED by this PEP but is expected to become feasible some time after successful implementation of the transition phases described in this PEP. It is expected that deprecation requires **a new process to deal with abandoned packages** because of unreachable maintainers for still popular packages. First transition phase (PyPI) ----------------------------- The proposed solution consists of multiple implementation and communication steps: #. Implement in PyPI the three modes described above, with an interface for package owners to select the mode for each package and register explicit external file URLs. #. For packages in all modes, label all links in the ``simple/`` index with ``rel="internal"`` or ``rel="external"``, to make it easier for client tools to distinguish the types of links in the second transition phase. #. Default all newly-registered packages to ``pypi-explicit`` mode (package owners can still switch to the other modes as desired). #. Determine (via an automated analysis tool) which packages have all installable files available on PyPI itself (group A), which have all installable files linked directly from PyPI metadata (group B), and which have installable versions available that are linked only from external homepage/download HTML pages (group C). #. Send mail to maintainers of projects in group A that their project will be automatically configured to ``pypi-explicit`` mode in one month, and similarly to maintainers of projects in group B that their project will be automatically configured to ``pypi-scrape`` mode. Inform them that this change is not expected to affect installability of their project at all, but will result in faster and safer installs for their users. Encourage them to set this mode themselves sooner to benefit their users. #. Send mail to maintainers of packages in group C that their package hosting mode is ``pypi-scrape-crawl``, list the URLs which currently are crawled, and suggest that they either re-host their packages directly on PyPI and switch to ``pypi-explicit``, or at least provide direct links to release files in PyPI metadata and switch to ``pypi-scrape``. Provide instructions and tools to help with these transitions. Second transition phase (installer tools) ----------------------------------------- For the second transition phase, maintainers of installation tools are asked to release two updates. The first update shall provide clear warnings if externally-hosted release files (that is, files whose link is ``rel="external"``) are selected for download, for which projects and URLs exactly this happens, and warn that in future versions externally-hosted downloads will be disabled by default. The second update should change the default mode to allow only installation of ``rel="internal"`` package files, and allow installation of externally-hosted packages only when the user supplies an option (ideally an option specifying exactly which external domains are to be trusted as download sources). When download of an externally-hosted package is disallowed, the user should be notified, with instructions for how to make the install succeed and warnings about the implication (that a file will be downloaded from a site that is not part of the package index). Open Questions / tasks ======================= - Should we introduce some form of PyPI API versioning in this PEP? (it might complicate matters and delay the implementation but is often seen as good practise). - do another round of discussions with installation tool authors and see about incorporating their feedback. There is one known issue in particular from Philip J. Eby who considers a host-based pattern matching algorithm preferable to interpreting "rel" attributes. References ========== .. [1] Donald Stufft, ratio of externally hosted versus pypi-hosted, http://mail.python.org/pipermail/catalog-sig/2013-March/005549.html (XXX need to update this data for all easy_install-supported formats) .. [2] Marc-Andre Lemburg, reasons for external hosting, http://mail.python.org/pipermail/catalog-sig/2013-March/005626.html .. [3] Holger Krekel, Script to remove homepage/download metadata for all releases http://mail.python.org/pipermail/catalog-sig/2013-February/005423.html Acknowledgments ================ Philip Eby for precise information and the basic ideas to implement the transition via server-side changes only. Donald Stufft for pushing away from external hosting and offering to implement both a Pull Request for the necessary PyPI changes and the analysis tool to drive the transition phase 1. Marc-Andre Lemburg, Nick Coghlan and catalog-sig in general for thinking through issues regarding getting rid of "external hosting". Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

1 0

Unify addresses passed to bind() and getaddrinfo()
by Giampaolo Rodolà March 14, 2013

March 14, 2013

>From http://docs.python.org/3/library/socket.html#example ...when using bind(): HOST = "" # Symbolic name meaning all available interfaces ...when using getaddrinfo(): HOST = None # Symbolic name meaning all available interfaces I recently got bitten by this difference in that I passed "" (empy string) to getaddrinfo() and since my DNS server was temporarily down getaddrinfo() hung indefinitely (cset where I fixed this: https://code.google.com/p/pyftpdlib/source/detail?r=1194#). It took me quite some time to figure out what was wrong as to me "" has always been an alias for "all interfaces" and didn't know getaddrinfo() behaved differently than bind() in this regard. If from getaddrinfo() standpoint "" (empty string) has no particular meaning I propose to unify the two APIs and make getaddrinfo() assume "" is an alias for "all interfaces". Thoughts? --- Giampaolo https://code.google.com/p/pyftpdlib/ https://code.google.com/p/psutil/ https://code.google.com/p/pysendfile/

2 2

Python 2.7 for Windows 9x
by André Gillibert March 13, 2013

March 13, 2013

Hello. Officially Python doesn't support Windows 9x and NT 3.51/4.0 anymore. I won't change this design decision. However, I built python 2.6.8 and 2.7.3 to run on these platforms. This may be useful to people with a very old computer. May you link this project on your python.org Web site? This is available at: <http://gillibert.fr/andre/projects/python/> Thanks. -- Sincerely André Gillibert

2 1

PEP XXX - Competitor with PEP 435: Adding an enum type to the Python standard library
by Ethan Furman March 13, 2013

March 13, 2013

First, I offer my apologies to all who are still battle-weary from the last flurry of enum threads. However, while flufl.enum (PEP 435) is a good package for the use-case it handles, there are plenty of use-cases that it does not handle, or doesn't handle well, that it should not be the implementation in the stdlib. To quote one of my earlier emails: > I'm beginning to see why enums as a class has not yet been added to Python. > We don't want to complicate the language with too many choices, yet there is > no One Obvious Enum to fit the wide variety of use-cases: > > - named int enums (http status codes) > - named str enums (tkinter options) > - named bitmask enums (file-type options) > - named valueless enums (any random set of names) > - named valueless-yet-orderable enums (any not-so-random set of names ;) This new PEP proposes an enum module that handles all those use cases, and makes it possible to handle others as well. If you recognize your idea but don't see your name in acknowledgements, please let me know. Code is available at https://bitbucket.org/stoneleaf/aenum -- ~Ethan~ ==============================================================================================

10 29

with ... except
by Antoine Pitrou March 10, 2013

March 10, 2013

Hello, A common pattern for me is to write a with statement for resource cleanup, but also handle specific errors after that. Right now, this is a bit cumbersome: try: with open("somefile", "rb)" as f: ... except FileNotFoundError: # do something else, perhaps actually create the file or: try: with transaction.commit_on_success(): ... except ObjectDoesNotExist: # do something else, perhaps clean up some internal cache How about adding syntax sugar for the above, in the form of a with ... except clause? It would nicely reduce spurious indentation, as with the try / except / finally which, long ago(!), helped reduce indentation and typing by removing the need to nest a try / except inside a try / finally. Regards Antoine.

10 18

Length hinting and preallocation for container types
by Christian Heimes March 9, 2013

March 9, 2013

Hello, today I came across this slides https://speakerdeck.com/alex/why-python-ruby-and-javascript-are-slow by Alex Gaynor. The slides aren't some random rants on Python. Alex makes some valid points. Coincidentally he is a PyPy developer, too. ;) One of his assertions is about memory (re)allocation. C is faster in some cases because C code usually does fewer allocations and reallocations. Python has no API to easily reallocate a list with 100 items. Code like lst = [] for i in range(100): lst.append(i*i) has to resize the list multiple times. PyPy has a feature to create a preallocated list. https://bitbucket.org/pypy/pypy/commits/2ff5e3c765ef/ Internally CPython already distinguishes between the length of object and the allocation size of an object for some types like list, dict and bytearray. For example PyListObject has `ob_size` for __len__ and `allocated` for the amount of available `ob_item` slots. I suggest that we add two new functions to container types like list, bytearray and dict. obj.__preallocate__(size) increases the internal buffer by size elements. obj.__shrink__() dwindles the internal buffer to len(obj) elements, maybe a bit more. A new context manager aids users with preallocation and shrinking: class LengthHint: def __init__(self, container, hint): self.container = container self.hint = hint self.instance = None def __enter__(self): self.instance = self.container() self.instance.__preallocate__(self.hint) return self.instance def __exit__(self, exc_type, exc_val, exc_tb): self.instance.__shrink__() with LengthHint(list, 200) as lst: # lst has 200 ob_item slots but len(lst) == 0 for i in range(100): lst.append(i*i) # __exit__ shrinks ob_item to 100 The C implementation is trivial as the three types already have all features. Christian

10 27

Official MySQL module
by Thomas Allen March 9, 2013

March 9, 2013

Hi, there! Do you plan to add an official module for connecting to MySQL databases? Existing third-party modules are bad-documented or no longer maintained... That's kinda strange, that such a nice language doesn't have it yet.

17 25

class-only methods without using metaclasses
by Eric Snow March 8, 2013

March 8, 2013

This tweet from Raymond helped distill something that was already on my mind of late: https://twitter.com/raymondh/status/309442149588533248 For some uses of @classmethod it makes sense to expose the methods on instances too. On others I just can't see it. In those cases, using @classmethod is justifiably a practical substitute for putting the methods on a metaclass. In my mind "alternate constructors" fall into this category. Would it be worth trying to get the best of both worlds (not exposed on instances but without metaclasses)? I can imagine providing a classmethod-like decorator that does this and have an implementation below. One benefit to not exposing the class-only methods on instances is that they don't clutter the instance namespace nor run the risk of colliding with instance-specific names. Thoughts? -eric ----------------------------------------------------------------------- class classonlymethod: """Like a classmethod but does not show up on instances. This is an alternative to putting the methods on a metaclass. It is especially meaningful for alternate constructors. """ # XXX or "metamethod" def __init__(self, method): self.method = method self.descr = classmethod(method) def __get__(self, obj, cls): name = self.method.__name__ getattr_static = inspect.getattr_static if obj is not None: # look up the attribute, but skip cls dummy = type(cls.__name__, cls.__bases__, {}) attr = getattr_static(dummy(), name, NOTSET) getter = getattr_static(attr, '__get__', None) # try data descriptors if (getter and getattr_static(attr, '__set__', False)): return getter(attr, obj, cls) # try the instance try: instance_dict = object.__getattribute__(obj, "__dict__") except AttributeError: pass else: try: return dict.__getitem__(instance_dict, name) except KeyError: pass # try non-data descriptors if getter is not None: return getter(attr, obj, cls) raise AttributeError(name) else: descr = vars(self)['descr'] return descr.__get__(obj, cls)

4 9

Re: [Python-ideas] Length hinting and preallocation for container types
by Eli Bendersky March 7, 2013

March 7, 2013

> > The real problem is that this code is not idiomatic Python, > especially if > > you want it to be reasonably fast: > > > > � lst = [] > > � for i in range(100): > > � � � lst.append(i*i) > > > > Why not: > > > > lst = [i*i for i in range(100)] > > > > If the "append" pattern is complex, just "preallocate" like this: > > > > lst = [0] * 100 > > > > And then fill it. > > > > Eli > > >How would you replicate the behavior of __exit__ in Christian's example? > > Why would I want to replicate it? Eli

5 4