[pypy-dev] new branch src-typedunwrap

[holger krekel]

On Fri, Jan 14, 2005 at 20:45 +0100, Christian Tismer wrote:
> Samuele Pedroni wrote:
> >After discussing with Armin and Holger, I have created a new branch to 
> >explore typed unwrapping.
> 
> Oh, did I miss an IRC discussion?

i guess so. We are actually discussing things on #pypy quite 
a bit these days. It would be great if you keep an eye on it :-) 

> Does somebody happen to have a log file of the session?
> I'm just curious to follow the reasoning, and to see
> if I can help something.

Sure, below you'll find the chat session.  I just cut the log at the 
start and the end but not in between.  Also there was an earlier 
(shorter) discussion which i didn't recover.  But almost all
of the arguments are recapped in the discussion below ... 

cheers, 

    holger


Jan 14 11:05:32 <pedronis>	should we also really rename is_true to true_w
Jan 14 11:05:43 <hpk>	IMO: yes 
Jan 14 11:05:55 <hpk>	we may do that in a branch as well 
Jan 14 11:06:05 <hpk>	and warn on pypy-dev 
Jan 14 11:06:10 <arigo>	good points
Jan 14 11:06:12 <pedronis>	I can create a branch
Jan 14 11:06:18 <hpk>	i don't know e.g. if Christian's translator breaks etc.pp. 
Jan 14 11:06:29 <hpk>	although global replace works fairly well i guess 
Jan 14 11:06:52 <hpk>	(but christian's translator is special because it targets interp-level python) 
Jan 14 11:07:14 <pedronis>	well, it also depends what we do with the generic unwrap
Jan 14 11:07:24 <pedronis>	if we remove it, rename it
Jan 14 11:07:54 <pedronis>	I think some other translator stuff may break and need changes
Jan 14 11:08:02 *	hpk is off for an hour to do tax declaration work (argl!) 
Jan 14 11:08:48 <arigo>	pedronis: I rediscovered unwrap_builtin()
Jan 14 11:08:56 <pedronis>	I saw that too
Jan 14 11:09:14 <arigo>	I'd believe that we only need that and the int_w&co
Jan 14 11:09:32 <pedronis>	probably yes
Jan 14 11:09:50 <pedronis>	I don't think we want to keep lists and tuple unwrappable
Jan 14 11:09:59 <arigo>	no it doesn't really make sense
Jan 14 11:09:59 <pedronis>	I mean directly unwrappable
Jan 14 11:10:17 <pedronis>	because of the mixed type cases
Jan 14 11:10:19 <arigo>	it's only used in some tests
Jan 14 11:10:36 <pedronis>	I think code objects use them
Jan 14 11:10:43 <pedronis>	to unwrap tuples of strings
Jan 14 11:10:50 <pedronis>	but it can be done differently
Jan 14 11:10:55 <arigo>	yes
Jan 14 11:11:30 <pedronis>	should str_w, int_w be multimethods in the std objspace
Jan 14 11:11:39 <pedronis>	if we consider multiple impl of types
Jan 14 11:11:44 <pedronis>	maybe yes
Jan 14 11:13:07 <arigo>	is_true() is no longer a MM
Jan 14 11:13:45 <pedronis>	yes, it's a bit of special case
Jan 14 11:13:56 <arigo>	right
Jan 14 11:14:03 <pedronis>	because it corresponds also to __nonzero__
Jan 14 11:14:11 <arigo>	well we have __int__, __str__ etc
Jan 14 11:14:12 <pedronis>	so it goes through descrops
Jan 14 11:14:30 <pedronis>	yes but I don't think we want int_w to correspond to __int__
Jan 14 11:14:38 <pedronis>	or maybe yes
Jan 14 11:14:41 <arigo>	I thought so
Jan 14 11:15:21 -->	lypanov (~alex at a80-126-190-213.adsl.xs4all.nl) has joined #pypy
Jan 14 11:15:47 <pedronis>	well but for str_w in often does not make sense, some function takes a string not everything that defines __str__
Jan 14 11:16:00 <pedronis>	s/takes/takes just
Jan 14 11:16:13 <arigo>	hum, right
Jan 14 11:16:35 <pedronis>	we already have space.int
Jan 14 11:16:37 <arigo>	should it be readbuf_w()?  (argh)
Jan 14 11:16:37 <pedronis>	btw
Jan 14 11:17:12 <pedronis>	that means that in parse_w we could have specifier both behaviors
Jan 14 11:17:25 <pedronis>	that menas something that implements the buffer intf
Jan 14 11:17:32 <pedronis>	s/menas/means
Jan 14 11:18:16 <arigo>	raah, trying to figure out what CPython really does is messy
Jan 14 11:18:29 <arigo>	suppose x.__class__ has an __int__
Jan 14 11:18:36 <arigo>	then you can do range(x) but not float(x)
Jan 14 11:18:55 <pedronis>	I think it is easier to have both kind of primitive op
Jan 14 11:18:58 <pedronis>	ops
Jan 14 11:19:17 <pedronis>	both unwrap only if of the "right" type or invoke the __xxx__ method
Jan 14 11:19:28 <pedronis>	and then be clever at parse_w level
Jan 14 11:19:31 <arigo>	(you can't do 1+x either, btw)
Jan 14 11:20:05 <pedronis>	yup
Jan 14 11:20:18 <arigo>	ouack
Jan 14 11:20:21 <pedronis>	so we need int_w which is different from int
Jan 14 11:20:29 <arigo>	"abcdef"[1:x] -> TypeError
Jan 14 11:20:33 <arigo>	"abcdef".__getslice__(1,x)
Jan 14 11:20:34 <arigo>	'bcdef'
Jan 14 11:21:17 <pedronis>	I think right now it is more worth to concetrate
Jan 14 11:21:31 <pedronis>	moving away from the generic unwrap to typed versions thereof
Jan 14 11:21:55 <pedronis>	sorting the mess what CPython accepts here and there
Jan 14 11:22:03 <pedronis>	is a slightly different issue
Jan 14 11:22:14 <arigo>	so we define int_w and str_w as only accepting the real type?
Jan 14 11:22:23 <pedronis>	yes
Jan 14 11:22:37 <pedronis>	you have space.int and space.str
Jan 14 11:22:43 <pedronis>	for the other behavior
Jan 14 11:23:18 <pedronis>	that's why I'm not sure that space.is_true should become true_w
Jan 14 11:23:24 <arigo>	I see
Jan 14 11:23:26 <pedronis>	true_w would just work with boolean
Jan 14 11:23:53 <pedronis>	although I think there's not much in CPython that behaves that way
Jan 14 11:24:04 <pedronis>	with bool args
Jan 14 11:24:53 <arigo>	it starts to look messy again, with a lot of xxx_w() MMs for each way we'd like to unwrap objects
Jan 14 11:25:04 <arigo>	readbuf_w(), writebuf_w(), ...?
Jan 14 11:25:36 <pedronis>	well, the othe option is to pass a 2nd type arg to unwrap
Jan 14 11:26:41 <pedronis>	but it gets tricky to express that as true RPython
Jan 14 11:26:53 -->	Eventh (evenwiik at v201a.studby.ntnu.no) has joined #pypy
Jan 14 11:27:05 <arigo>	float_w should accept an int, I guess
Jan 14 11:28:04 <pedronis>	well, if they are multimethods at least in the std objspace
Jan 14 11:28:10 <pedronis>	it easy to tweak that
Jan 14 11:28:49 <arigo>	I'm more trying to understand what we really want...
Jan 14 11:30:06 <pedronis>	right now a big parts of our unwraps
Jan 14 11:30:12 <pedronis>	are about ints and strings
Jan 14 11:31:01 <pedronis>	so at least int_w and str_w seem to make sense
Jan 14 11:31:35 <arigo>	ok
Jan 14 11:31:54 <pedronis>	parse_w will also make sense
Jan 14 11:32:10 <pedronis>	so I think creating a branch
Jan 14 11:32:26 <pedronis>	and introducing int_w and str_w as first step
Jan 14 11:32:46 <arigo>	looks good
Jan 14 11:32:58 <pedronis>	then I think it is easier to see what we need beyond that
Jan 14 11:33:04 <pedronis>	from parse_w point of view
Jan 14 11:33:12 <arigo>	ok.
Jan 14 11:36:38 <pedronis>	I will create a src-typedunwrap branch
Jan 14 11:36:43 <pedronis>	and mail pypy-dev
Jan 14 11:36:57 <pedronis>	now I'm going to have lunch
Jan 14 11:37:03 ---	pedronis is now known as pedronis_lunch
Jan 14 11:55:42 <--	arigo has quit ("Leaving")
Jan 14 12:15:09 <--	thingie24 has quit (leguin.freenode.net irc.freenode.net)
Jan 14 12:15:09 <--	dlk has quit (leguin.freenode.net irc.freenode.net)
Jan 14 12:15:09 <--	etrepum has quit (leguin.freenode.net irc.freenode.net)
Jan 14 12:20:20 -->	thingie24 (~rmt38 at valhalla.ccp.cc) has joined #pypy
Jan 14 12:20:20 -->	dlk (~dlk at walter.ita.chalmers.se) has joined #pypy
Jan 14 12:20:20 -->	etrepum (bob at ayunami.redivi.com) has joined #pypy
Jan 14 12:22:56 -->	arigo (~arigo at d83-176-50-66.cust.tele2.ch) has joined #pypy
Jan 14 12:26:30 ---	pedronis_lunch is now known as pedronis
Jan 14 12:47:44 <--	adim (~adim at logilab.net2.nerim.net) has left #pypy
Jan 14 12:57:18 <mwh>	afternoon
Jan 14 12:59:42 <pedronis>	hi
Jan 14 12:59:57 <arigo>	afternoon
Jan 14 13:08:07 <hpk>	afternoon
Jan 14 13:22:27 <lac-sleeps>	afternoon
Jan 14 13:22:32 ---	lac-sleeps is now known as lac
Jan 14 13:27:54 <mwh>	nothing else to say, i'm afraid :)
Jan 14 13:31:46 <lac>	:-)
Jan 14 14:05:57 <mwh>	arigo: i hope people read and take notice of your python-dev interface post
Jan 14 14:07:54 <arigo>	i hesitated
Jan 14 14:08:08 <arigo>	it's a bit lost in the thread but i don't want to insist more, specially to avoid flooding python-dev again
Jan 14 14:10:38 <mwh>	well, i think the idea that strings represent >1 concept is illuminating
Jan 14 14:14:32 -->	adim (~adim at logilab.net2.nerim.net) has joined #pypy
Jan 14 14:59:59 <lac>	I am going to work now. With a bottle of champaigne.
Jan 14 15:00:04 <mwh>	hehe
Jan 14 15:00:05 ---	lac is now known as lac-work
Jan 14 15:12:19 <hpk>	arigo, mwh: indeed i find "global adapter registries" scary
Jan 14 15:14:35 <hpk>	another serious concern which i haven't seen mentioned is debugging, btw 
Jan 14 15:15:16 <hpk>	for example, how will a traceback involving adapations on arguments look like? 
Jan 14 15:21:29 <arigo>	if there is an exception while adapting?
Jan 14 15:22:17 <arigo>	you'll probably see the "def f(x: y)" line in the traceback, which is a good hint that it's during adaptation
Jan 14 15:34:57 <--	idnar has quit ("kbye")
Jan 14 16:07:41 <hpk>	i am pretty sure that it will make things harder to debug, anyway 
Jan 14 16:15:37 <--	dlk has quit (Read error: 104 (Connection reset by peer))
Jan 14 16:23:31 <arigo>	the global registry is dangerous
Jan 14 16:23:49 <arigo>	__conform__() and __adapt__() should be reasonable, though
Jan 14 16:24:21 <hpk>	as methods on objects? (i didn't follow python-dev closely on all this)
Jan 14 16:24:55 <arigo>	I think the idea is that the built-in adapt() function just calls __conform__() and __adapt__()
Jan 14 16:25:13 <arigo>	a bit like "a+b" calls __add__() and __radd__(), btw
Jan 14 16:25:57 <pedronis>	arigo: unrelated, I'm thinking what to do about co_consts
Jan 14 16:26:33 <pedronis>	I rediscovered the fact that our code objects are supposed to be space indipendent
Jan 14 16:26:48 <pedronis>	but that means they must store co_consts unwrapped
Jan 14 16:27:00 <arigo>	yes
Jan 14 16:27:31 <pedronis>	but that exactly a mixed type tuple
Jan 14 16:27:49 <arigo>	oups
Jan 14 16:28:11 <pedronis>	so we would like co_consts_w
Jan 14 16:28:34 <pedronis>	but the the code objects are really designed to be space indipendent
Jan 14 16:29:15 <hpk>	although i am not sure this design decision is neccessary 
Jan 14 16:30:15 <hpk>	what does it buy us? 
Jan 14 16:30:16 <pedronis>	probably not, but is is quite pervasive
Jan 14 16:30:28 <hpk>	yes, and Armin had good arguments 
Jan 14 16:31:24 <pedronis>	because the gateways are also space independent
Jan 14 16:31:47 <arigo>	it's all a bit messy
Jan 14 16:32:26 <pedronis>	I have converted most of interpreter/* to use just int_w,str_w and unwrap_builtin
Jan 14 16:32:54 <arigo>	there are really two approaches that exist in practice:
Jan 14 16:32:59 <pedronis>	althoug unwrap_builtin could probably have a better name
Jan 14 16:33:05 <arigo>	in CPython we have real objects in co_consts
Jan 14 16:33:30 <arigo>	in many other VMs the literals are stored in some bytecodeish compact format
Jan 14 16:33:46 <arigo>	and expanded into real objects by the equivalent of LOAD_CONST
Jan 14 16:34:05 <arigo>	e.g. LOAD_STRING n, where n somehow points to raw string data
Jan 14 16:34:32 <arigo>	so far PyPy does the latter, with LOAD_CONST n meaning wrap the "raw" co_consts[n]
Jan 14 16:35:22 <arigo>	a the nice property is that we don't have to preprocess code objects: the interpreter could theoretically just run off mmaped .pyc files
Jan 14 16:36:20 <pedronis>	ah but then our co_const would be in marshal format
Jan 14 16:37:11 <arigo>	yes, note that the marhsal format is similar to whatever format we'd have to invent to represent a mixed type tuple anyway
Jan 14 16:38:09 <pedronis>	well, for now I keep the unwrap and unwrap
Jan 14 16:38:33 <arigo>	in theory, we could tell the translator that the Code class and its attributes are implemented as a "view" over read-only packed data in marshal format
Jan 14 16:38:45 <pedronis>	and *if* we remove direct tuple list wrapping/unwrapping
Jan 14 16:38:52 <arigo>	and get a version of the interpreter that really handles only marshal data, never "real" Code instances
Jan 14 16:39:18 <pedronis>	yes
Jan 14 16:39:58 <hpk>	probably 
Jan 14 16:40:12 <arigo>	well it's messy anyway, because of long or complex literals that are maybe not R-Python types
Jan 14 16:40:38 <arigo>	I mean, literals of type long or complex
Jan 14 16:41:33 <pedronis>	also true
Jan 14 16:41:50 <pedronis>	well with marshal approach
Jan 14 16:42:20 <pedronis>	that's not an issue
Jan 14 16:42:37 <pedronis>	but those wrap and unwrap need to be spelled in a special way
Jan 14 16:43:13 <arigo>	hum, it looks like a marshaled object is an object is the MarshalObjSpace :-)
Jan 14 16:43:35 <pedronis>	well that's still a special way
Jan 14 16:43:40 <arigo>	then those wrap and unwrap become sending objects across spaces
Jan 14 16:43:42 <pedronis>	because it's a different space
Jan 14 16:43:49 <arigo>	precisely.
Jan 14 16:43:55 <mwh>	arigo: i suspect you might be getting ahead of yourself here?
Jan 14 16:44:02 <pedronis>	:)
Jan 14 16:44:13 <hpk>	oh, he is probably thinking of making use of adapt() :-) 
Jan 14 16:44:26 <hpk>	(not really) 
Jan 14 16:44:33 <pedronis>	for now I will leave the wrap and unwrap
Jan 14 16:44:36 <arigo>	hpk: not that I am fully aware of :-)
Jan 14 16:44:39 <mwh>	adapt(1, ApplicationLevel) ?
Jan 14 16:44:40 <pedronis>	anyway I have not disabled
Jan 14 16:44:52 <mwh>	adapt(1, IApplicationLevelThingie), rather
Jan 14 16:44:56 <pedronis>	the wrap/unwrap functionality for tuples or lists
Jan 14 16:45:00 <pedronis>	if I do that
Jan 14 16:45:01 <mwh>	let's use a decorator too!
Jan 14 16:45:08 <arigo>	pedronis: ok
Jan 14 16:45:15 <pedronis>	I will use some special names for those cases
Jan 14 16:45:18 <arigo>	mwh: :-)
Jan 14 16:47:08 <arigo>	well it's all fuzzy, e.g. what should a built-in compiler produce?  Code objects with what in the co_consts?  It looks like it shouldn't depend on a space...
Jan 14 16:48:01 <mwh>	well, indeed
Jan 14 16:49:28 <pedronis>	yes, but I don't see a problem in finding a suitable space independent format for co_consts
Jan 14 16:49:51 <hpk>	the fact that we want to have a space-independent repr of a code object does not imply that PyPy's internal code objects could not be space-dependent
Jan 14 16:50:15 <hpk>	(sorry for the negation-complexity here :^) 
Jan 14 16:52:54 <arigo>	ok
Jan 14 16:53:25 <arigo>	that's quite possible after all...
Jan 14 16:54:15 <hpk>	also it seems that apart from LOAD_CONST we hardly have "generic" wraps in interpreter/
Jan 14 16:54:43 <pedronis>	yup
Jan 14 16:54:44 <mwh>	most of them are wraps of obvious strings or integers, i think
Jan 14 16:55:01 <hpk>	yes, mostly wraps of messages, filenames, varnames
Jan 14 16:55:35 <pedronis>	as I said the generic unwrap is also the one of co_consts
Jan 14 16:55:42 <pedronis>	s/the/the only
Jan 14 16:56:43 <arigo>	there must be a generic wrap for default args of gateways
Jan 14 16:57:53 <pedronis>	yes, I said unwrap
Jan 14 16:58:22 <hpk>	app2interp.getdefaults() does the generic wrap, right 
Jan 14 16:58:28 <pedronis>	it's also marked as NOT_RPYTHON
Jan 14 16:58:33 <hpk>	yes :-) 
Jan 14 16:58:36 <pedronis>	and happens at bootstrap
Jan 14 16:58:47 <arigo>	ok ok :-)
Jan 14 17:00:56 <arigo>	if we want to allow things like  new.code(...., co_consts=(AnyThing(),), ...)
Jan 14 17:01:03 <arigo>	to be done by the user
Jan 14 17:01:13 <arigo>	then we have no choice but keep the object wrapped
Jan 14 17:01:37 <pedronis>	true
Jan 14 17:03:16 <hpk>	the same regarding func_defaults 
Jan 14 17:03:25 <hpk>	it works in CPython (to override func_defaults) 
Jan 14 17:03:28 <hpk>	but doesn't in PyPy apparently 
Jan 14 17:04:15 <arigo>	no?
Jan 14 17:04:39 <arigo>	you can say   def f(x=AnyThing())
Jan 14 17:04:48 <arigo>	so you can create func_defaults with anything at all
Jan 14 17:05:01 <hpk>	yes, maybe the current limitation is just shallow 
Jan 14 17:05:02 <arigo>	(even though we might not support assigning to .func_defaults)
Jan 14 17:05:19 <pedronis>	Functions are space depedent
Jan 14 17:05:21 <arigo>	that's quite different, func_defaults are not a syntactic property
Jan 14 17:05:31 <pedronis>	in fact they store defaults as defs_w
Jan 14 17:06:07 <pedronis>	it's code objects and gateways that are space independent
Jan 14 17:10:54 <pedronis>	looking further maybe it's not too hard to make code objects space depedent if we want to
Jan 14 17:11:42 <pedronis>	it seems gateways just need to become even more lazy
Jan 14 17:12:26 <arigo>	could BuiltinCode remain space-independent, and PyCode be space-dependent?
Jan 14 17:13:53 <pedronis>	I think so
Jan 14 17:14:08 <hpk>	it should be possible
Jan 14 17:15:00 <pedronis>	BuiltinCode is constructing a PyCode just to be able to use .signature
Jan 14 17:15:15 <pedronis>	but can be done in some other way
Jan 14 17:15:35 <hpk>	i guess we should untangle the cases more
Jan 14 17:15:55 <hpk>	maybe we tried too hard to reuse code for BuiltinCode/PyCode cases 
Jan 14 17:17:37 <pedronis>	the question is, do we want PyCode to be space depedent, it probably has to because of what we found
Jan 14 17:18:19 <arigo>	I guess so
Jan 14 17:18:43 <pedronis>	I can do the change
Jan 14 17:19:16 <pedronis>	on the branch?
Jan 14 17:19:31 <hpk>	makes sense i think 
Jan 14 17:20:27 <pedronis>	I'm going first to check in what I did so far
Jan 14 17:23:48 <arigo>	ok :-)