<meta http-equiv="content-type" content="text/html; charset=utf-8"><div><br></div><div>I'm a unix guy. That's what we call a sort-uniq operation, after the pipeline we'd use: sort datafile | uniq > uniq-lines.txt. So I google that with python and ....</div>
<div><br></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8">As Jason Petrone wrote when he withdrew PEP 270 in <a href="http://www.python.org/dev/peps/pep-0270/">http://www.python.org/dev/peps/pep-0270/</a>:</div>
<meta http-equiv="content-type" content="text/html; charset=utf-8"><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"></blockquote><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><br></div><div>"creating a sequence without duplicates is just a matter of</div><div>choosing a different data structure: a set instead of a list."</div></blockquote><div><br></div><div>At the time, sets.py was a nifty new thing. Since then, the set datatype has</div>
<div>been added to python's base.</div><div><br></div><div>set() can consume a list of tuples, but not a list of lists, like the X you showed us. You're job will be getting your massive list of lists into a list of tuples.</div>
<div><br></div><div>This works, but for your very large arrays, may take large time:</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><font class="Apple-style-span" face="'courier new', monospace">X = [[1,2], [1,2], [3,4], [3,4]]</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div><div><font class="Apple-style-span" face="'courier new', monospace">Y = set( [tuple(x) for x in X] )</font></div></blockquote>
<div><br></div><div>There may be faster methods. The map() function might help, but I really don't know. Here's something to try:</div><div><span class="Apple-style-span" style="font-family: 'courier new', monospace; "><br>
</span></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><span class="Apple-style-span" style="font-family: 'courier new', monospace; ">Y = set( map(tuple, X )</span></div>
</blockquote><div><br></div><div>Or you can go old school route, from before the days of set(), that is:</div><meta http-equiv="content-type" content="text/html; charset=utf-8"><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<a href="http://code.activestate.com/recipes/52560-remove-duplicates-from-a-sequence/">http://code.activestate.com/recipes/52560-remove-duplicates-from-a-sequence/</a></blockquote><div><br></div><div>Best,</div><div>Mike</div>
<br><div class="gmail_quote">On Wed, Dec 22, 2010 at 10:28 AM, Aahz <span dir="ltr"><<a href="mailto:aahz@pythoncraft.com">aahz@pythoncraft.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On Wed, Dec 22, 2010, otrov wrote:<br>
><br>
> I failed in my first idea to solve this problem with matlab/octave,<br>
> as I just started using this tools for data manipulation, and then<br>
> thought to try python as more feature rich descriptive language and<br>
> post this problem to python group I'm subscribed already<br>
<br>
</div>You may get better answers posting to a general Python group (e.g.<br>
comp.lang.python).<br>
<font color="#888888">--<br>
Aahz (<a href="mailto:aahz@pythoncraft.com">aahz@pythoncraft.com</a>) <*> <a href="http://www.pythoncraft.com/" target="_blank">http://www.pythoncraft.com/</a><br>
<br>
"Think of it as evolution in action." --Tony Rand<br>
</font><div><div></div><div class="h5">_______________________________________________<br>
python-win32 mailing list<br>
<a href="mailto:python-win32@python.org">python-win32@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/python-win32" target="_blank">http://mail.python.org/mailman/listinfo/python-win32</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Mike Diehn<br>Senior Systems Administrator<br>ANSYS, Inc - Lebanon, NH Office<br><a href="mailto:mike.diehn@ansys.com">mike.diehn@ansys.com</a>, (603) 727-5492<br>