Iterating over several lists at once
Question posted by: Gal Diskin
(Guest)
on
December 13th, 2006 12:25 PM
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...
I was wondering if one could write this more easily in some manner
using only 1 for loop.
What I mean is something like this:
for (x1,x2,x3) in (l1,l2,l3):
print "do something with", x1, x2, x3
Or maybe like this:
for x1 in l1, x2 in l2, x3 in l3:
print "do something with", x1, x2, x3
However, this code obviously doesn't work...
I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).
Thanks in advance,
Gal
Would you like to answer this question?
Sign up for a free account, or Login (if you're already a member).
|
|
December 13th, 2006 12:35 PM
# 2
|
Re: Iterating over several lists at once
"Gal Diskin" wrote:
Quote:
Originally Posted by
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
>
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
>
What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...
>
I was wondering if one could write this more easily in some manner
using only 1 for loop.
What I mean is something like this:
>
for (x1,x2,x3) in (l1,l2,l3):
print "do something with", x1, x2, x3
|
how about
for x1, x2, x3 in func(l1, l2, l3):
print x1, x2, x3
where func is defined as, say,
def func(l1, l2, l3):
return ((x1, x2, x3) for x1 in l1 for x2 in l2 for x3 in l3)
or if you prefer
def helper(l1, l2, l3):
for x1 in l1:
for x2 in l2:
for x3 in l3:
yield x1, x2, x3
</F>
|
|
|
December 13th, 2006 12:35 PM
# 3
|
Re: Iterating over several lists at once
Gal Diskin wrote:
Quote:
Originally Posted by
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
>
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
|
What's wrong with this?
[...]
Quote:
Originally Posted by
I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).
|
def cartesian_product(l1, l2, l3):
for i in l1:
for j in l2:
for k in l3:
yield (i, j, k)
for (i, j, k) in cartesian_product(l1, l2, l3):
print "do something with", i, j, k
--
Roberto Bonvallet
|
|
|
December 13th, 2006 01:15 PM
# 4
|
Re: Iterating over several lists at once
"Gal Diskin" <gal.diskin@gmail.comwrites:
Quote:
Originally Posted by
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
>
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
|
This does look a little kludgy (untested):
for x1,x2,x3 in ((x1,x2,x3) for x1 in l1 for x2 in l2 for x3 in l3):
print "do something with", x1, x2, x3
|
|
|
December 13th, 2006 01:25 PM
# 5
|
Re: Iterating over several lists at once
Gal Diskin wrote:
Quote:
Originally Posted by
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
|
Quote:
Originally Posted by
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
|
Quote:
Originally Posted by
I was wondering if one could write this more easily in some manner
using only 1 for loop.
|
def nested_loops(*args):
assert args
if len(args) == 1:
for item in args[0]:
yield (item,)
else:
gap = len(args) // 2
for left in nested_loops(*args[:gap]):
for right in nested_loops(*args[gap:]):
yield left + right
if __name__ == "__main__":
for i, k in nested_loops("abc", "12"):
print i, k
for i, j, k in nested_loops("ab", "123", "xyz"):
print i, j, k
Be prepared for a significant performance hit.
Peter
PS: Did anybody say macro? No? I must be hallucinating...
|
|
|
December 13th, 2006 01:25 PM
# 6
|
Re: Iterating over several lists at once
Nothing seriously wrong, but it's not too elegent. Especially when the
number of lists you want to iterate over gets bigger (especially
because of the indentation in python). As you noticed (an phrased
better than me), what I was wondering is if there is a way to iterate
over the cartesian product, but without actually doing all n for loops
but using a single "for" loop.
Thanks for replying me.
On Dec 13, 3:58 pm, Roberto Bonvallet <Roberto.Bonval...@cern.ch>
wrote:
Quote:
Originally Posted by
Gal Diskin wrote:
Quote:
Originally Posted by
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
|
>
Quote:
Originally Posted by
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3What's wrong with this?
|
>
[...]
>
Quote:
Originally Posted by
I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).def cartesian_product(l1, l2, l3):
|
for i in l1:
for j in l2:
for k in l3:
yield (i, j, k)
>
for (i, j, k) in cartesian_product(l1, l2, l3):
print "do something with", i, j, k
>
--
Roberto Bonvallet
|
|
|
|
December 13th, 2006 01:35 PM
# 7
|
Re: Iterating over several lists at once
Gal Diskin schrieb:
Quote:
Originally Posted by
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
>
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
>
What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...
|
Heard about recursion?
def collect(L,*lists):
if not lists:
return [(x,) for x in L]
collection = []
for x in L:
for y in collect(lists[0],*lists[1:]):
collection.append((x,)+y)
return collection
for item in collect( l1, l2, l3):
func(*item)
Here is the same in generator form
def collect(L,*lists):
if not lists:
for x in L:
yield (x,)
else:
for x in L:
for y in collect(lists[0],*lists[1:]):
yield (x,)+y
( o.k - it required two nested for-loops in each implementation :)
|
|
|
December 13th, 2006 01:35 PM
# 8
|
Re: Iterating over several lists at once
Thanks, that's an improvment (your first way).
But I still wish I could find an even shorter (or more elegent) way of
doing it. (Well, I guess if I expect every wish I have to come true I
should at least wish for something more valuable.)
Thanks again,
Gal
On Dec 13, 3:58 pm, "Fredrik Lundh" <fred...@pythonware.comwrote:
Quote:
Originally Posted by
"Gal Diskin" wrote:
Quote:
Originally Posted by
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
|
>
Quote:
Originally Posted by
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
|
>
Quote:
Originally Posted by
What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...
|
>
Quote:
Originally Posted by
I was wondering if one could write this more easily in some manner
using only 1 for loop.
What I mean is something like this:
|
>
Quote:
Originally Posted by
for (x1,x2,x3) in (l1,l2,l3):
print "do something with", x1, x2, x3how about
|
>
for x1, x2, x3 in func(l1, l2, l3):
print x1, x2, x3
>
where func is defined as, say,
>
def func(l1, l2, l3):
return ((x1, x2, x3) for x1 in l1 for x2 in l2 for x3 in l3)
>
or if you prefer
>
def helper(l1, l2, l3):
for x1 in l1:
for x2 in l2:
for x3 in l3:
yield x1, x2, x3
>
</F>
|
|
|
|
December 13th, 2006 03:25 PM
# 9
|
Re: Iterating over several lists at once
Hi,
if you "needs to iterate over 3 lists at the same" and according your example
Quote:
Originally Posted by
for (x1,x2,x3) in (l1,l2,l3):
print "do something with", x1, x2, x3
|
i has guessed that you need this (may be i was wrong):
a = (1,2,3, 1)
b = (4,5,6)
c = (7,8,9, 2, 3)
for x, y, z in zip(a, b, c):
print x, y, z
or this
for x, y, z in map(None, a, b, c):
print x,y,z
Try both examples with tuples that have various length, they have difference
Gal Diskin wrote:
Quote:
Originally Posted by
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
>
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
>
What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...
>
>
I was wondering if one could write this more easily in some manner
using only 1 for loop.
What I mean is something like this:
>
for (x1,x2,x3) in (l1,l2,l3):
print "do something with", x1, x2, x3
>
Or maybe like this:
>
for x1 in l1, x2 in l2, x3 in l3:
print "do something with", x1, x2, x3
>
However, this code obviously doesn't work...
>
>
I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).
>
Thanks in advance,
Gal
>
|
--
Maksim Kasimov
|
|
|
December 13th, 2006 05:15 PM
# 10
|
Re: Iterating over several lists at once
Gal Diskin wrote:
Quote:
Originally Posted by
On Dec 13, 3:58 pm, Roberto Bonvallet <Roberto.Bonval...@cern.ch>
wrote:
Quote:
Originally Posted by
Gal Diskin wrote:
Quote:
Originally Posted by
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
|
Quote:
Originally Posted by
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3What's wrong with this?
|
[...]
Quote:
Originally Posted by
I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).def cartesian_product(l1, l2, l3):
|
for i in l1:
for j in l2:
for k in l3:
yield (i, j, k)
for (i, j, k) in cartesian_product(l1, l2, l3):
print "do something with", i, j, k
|
>
Nothing seriously wrong, but it's not too elegent.
|
I wonder why you think that. Many people here would consider it the
height of elegance to take some complicated logic, writing a function
encompassing the complicated parts, and ending up with simpler logic.
That's what happens here.
The cartesian_product solution above is not much longer than this
syntax you proposed:
for (i,j,k) in (a,b,c):
use(i,j,k)
(You can make the function name shorter if you want.) What is it that
you find inelegant about this solution? Is it that you don't like
extra function sitting there? Just put it in a module and import it.
Quote:
Originally Posted by
Especially when the
number of lists you want to iterate over gets bigger (especially
because of the indentation in python).
|
The function can be extended to allow arbitrary arguments. Here's a
non-minmal recursive version.
def cartesian_product(*args):
if len(args) 1:
for item in args[0]:
for rest in cartesian_product(*args[1:]):
yield (item,) + rest
elif len(args) == 1:
for item in args[0]:
yield (item,)
else:
yield ()
Quote:
Originally Posted by
As you noticed (an phrased
better than me), what I was wondering is if there is a way to iterate
over the cartesian product, but without actually doing all n for loops
but using a single "for" loop.
|
Even if Python had a special syntax for it, it would still be looping
internally.
And there's almost no chance of there ever being a special syntax for
it. First of all, it's very straightforward to do with functional
solutions, as we've shown here. Second, compare the case of zip.
Iteration in parallel (such as you would do with zip) is a lot more
common than nested iteration (cartesian_product), but even parallel
iteration doesn't have a special syntax, and th BDFL had declared that
won't in Python 3.0.
The best you could hope for is a built in function like zip, and that's
doubtful. OTOH, something like this has decent shot of appearing in a
standard functional or iterator module of some sort.
Quote:
Originally Posted by
Thanks for replying me.
|
You're welcome in advance. Also, one note about comp.lang.python
and/or python-list: it's conventional here to put replies beneath the
quoted text. This is for the benefit of future readers, so they don't
have read conversations in reverse order.
Carl Banks
|
|
|
December 13th, 2006 05:35 PM
# 11
|
Re: Iterating over several lists at once
Sorry for breaking into this thread, but I agree completely that any
unnecessary indentations should be avoided. For the same reason I advocate
that the following syntax should work:
for x in some_list if some_condition:
... code ...
in stead of
for x in some_list
if some_condition:
... code ...
All the best!
@
PS: maybe using 'sets' can help you out for a particular problem.
Gal Diskin wrote:
Quote:
Originally Posted by
Nothing seriously wrong, but it's not too elegent. Especially when the
number of lists you want to iterate over gets bigger (especially
because of the indentation in python). As you noticed (an phrased
better than me), what I was wondering is if there is a way to iterate
over the cartesian product, but without actually doing all n for loops
but using a single "for" loop.
>
Thanks for replying me.
>
>
On Dec 13, 3:58 pm, Roberto Bonvallet <Roberto.Bonval...@cern.ch>
wrote:
Quote:
Originally Posted by
>Gal Diskin wrote:
Quote:
Originally Posted by
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
|
>>
Quote:
Originally Posted by
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3What's wrong with this?
|
>>
>[...]
>>
Quote:
Originally Posted by
I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).def
cartesian_product(l1, l2, l3):
|
> for i in l1:
> for j in l2:
> for k in l3:
> yield (i, j, k)
>>
>for (i, j, k) in cartesian_product(l1, l2, l3):
> print "do something with", i, j, k
>>
>--
>Roberto Bonvallet
|
|
|
|
|
December 14th, 2006 09:25 PM
# 12
|
Re: Iterating over several lists at once
* at (at@tuko.nl) wrote:
Quote:
Originally Posted by
Sorry for breaking into this thread, but I agree completely that any
unnecessary indentations should be avoided. For the same reason I advocate
that the following syntax should work:
>
for x in some_list if some_condition:
... code ...
>
in stead of
>
for x in some_list
if some_condition:
... code ...
|
It is possible to avoid the extra level of indentaion, but I think it's
much less readable than the 2-level verbose expresion:
[1, 2, 3, 4, 5, 6, 7]
Quote:
Originally Posted by
Quote:
Originally Posted by
Quote:
Originally Posted by
>>for odd in (num for num in a if num % 2 == 1):
|
|
|
.... print odd
....
1
3
5
7
there is also continue, which I think is a good compromise:
Quote:
Originally Posted by
Quote:
Originally Posted by
Quote:
Originally Posted by
>>for num in a:
|
|
|
.... if num % 2 == 0:
.... continue
.... print num
....
1
3
5
7
HTH (and not lead astray),
mike
|
|
|
December 14th, 2006 09:25 PM
# 13
|
Re: Iterating over several lists at once
Carl Banks wrote:
<snip>
Quote:
Originally Posted by
>
The function can be extended to allow arbitrary arguments. Here's a
non-minmal recursive version.
>
def cartesian_product(*args):
if len(args) 1:
for item in args[0]:
for rest in cartesian_product(*args[1:]):
yield (item,) + rest
elif len(args) == 1:
for item in args[0]:
yield (item,)
else:
yield ()
>
>
|
Very nice.
|
|
|
December 14th, 2006 09:25 PM
# 14
|
Re: Iterating over several lists at once
John Henry wrote:
Quote:
Originally Posted by
Carl Banks wrote:
<snip>
Quote:
Originally Posted by
>The function can be extended to allow arbitrary arguments. Here's a
>non-minmal recursive version.
>>
>def cartesian_product(*args):
> if len(args) 1:
> for item in args[0]:
> for rest in cartesian_product(*args[1:]):
> yield (item,) + rest
> elif len(args) == 1:
> for item in args[0]:
> yield (item,)
> else:
> yield ()
>>
>>
|
>
Very nice.
>
|
another implementation of cartesian_product using 'reduce' to add
mystery ;-)
def star(outer, inner):
for rest in outer:
for item in inner:
yield rest + (item,)
def cartesian_product(*args):
return reduce(star, args, ((),))
Quote:
Originally Posted by
Quote:
Originally Posted by
Quote:
Originally Posted by
>>list(cartesian_product("01","01","01"))
|
|
|
[('0', '0', '0'), ('0', '0', '1'), ('0', '1', '0'), ('0', '1', '1'),
('1', '0', '0'), ('1', '0', '1'), ('1', '1', '0'), ('1', '1', '1')]
Michael
|
|
|
December 14th, 2006 09:26 PM
# 15
|
Re: Iterating over several lists at once
Is this suppose to be a brain teaser or something?
Michael Spencer wrote:
Quote:
Originally Posted by
John Henry wrote:
Quote:
Originally Posted by
Carl Banks wrote:
<snip>
Quote:
Originally Posted by
The function can be extended to allow arbitrary arguments. Here's a
non-minmal recursive version.
>
def cartesian_product(*args):
if len(args) 1:
for item in args[0]:
for rest in cartesian_product(*args[1:]):
yield (item,) + rest
elif len(args) == 1:
for item in args[0]:
yield (item,)
else:
yield ()
>
>
|
Very nice.
|
another implementation of cartesian_product using 'reduce' to add
mystery ;-)
>
def star(outer, inner):
for rest in outer:
for item in inner:
yield rest + (item,)
>
def cartesian_product(*args):
return reduce(star, args, ((),))
>
Quote:
Originally Posted by
Quote:
Originally Posted by
>>list(cartesian_product("01","01","01"))
|
|
[('0', '0', '0'), ('0', '0', '1'), ('0', '1', '0'), ('0', '1', '1'),
('1', '0', '0'), ('1', '0', '1'), ('1', '1', '0'), ('1', '1', '1')]
>
Michael
|
|
|
|
December 27th, 2006 11:05 AM
# 16
|
Re: Iterating over several lists at once
On Dec 13, 3:47 pm, "Gal Diskin" <gal.dis...@gmail.comwrote:
Quote:
Originally Posted by
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
>
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
>
What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...
>
I was wondering if one could write this more easily in some manner
using only 1 for loop.
What I mean is something like this:
>
for (x1,x2,x3) in (l1,l2,l3):
print "do something with", x1, x2, x3
>
Or maybe like this:
>
for x1 in l1, x2 in l2, x3 in l3:
print "do something with", x1, x2, x3
>
However, this code obviously doesn't work...
>
I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).
>
Thanks in advance,Gal
|
Sorry for bumping this thread up. I just got back from a vacation and I
had to say thanks to all the people that took the time to answer me.
To Maksim - that's not what I meant, I want to iterate over the
cartesian product of the lists (all ordered tuples, _not_ the tuple
created from the first item in each list, then the tubple created from
the second item in each list... as your example does).
Thanks,
Gal
|
|
|
December 27th, 2006 06:45 PM
# 17
|
Re: Iterating over several lists at once
"Gal Diskin" <gal.diskin@gmail.comwrote in message
news:1167220526.849221.252410@f1g2000cwa.googlegro ups.com...
Quote:
Originally Posted by
>
>
On Dec 13, 3:47 pm, "Gal Diskin" <gal.dis...@gmail.comwrote:
Quote:
Originally Posted by
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...
|
|
I don't see your previous article. Not sure if you are still looking for
a solution, but here's one:
Quote:
Originally Posted by
Quote:
Originally Posted by
Quote:
Originally Posted by
>>[(x, y, z) for x in [1,2,3] for y in list('abc') for z in list('XY')]
|
|
|
[(1, 'a', 'X'), (1, 'a', 'Y'), (1, 'b', 'X'), (1, 'b', 'Y'), (1, 'c', 'X'),
(1, 'c', 'Y'), (2, 'a', 'X'), (2, 'a', 'Y'), (2, 'b', 'X'), (2, 'b', 'Y'),
(2, 'c', 'X'), (2, 'c', 'Y'), (3, 'a', 'X'), (3, 'a', 'Y'), (3, 'b', 'X'),
(3, 'b', 'Y'), (3, 'c', 'X'), (3, 'c', 'Y')]
This is a bit more compact, but I don't see anything wrong with what you
have.
|
|
|
December 27th, 2006 06:55 PM
# 18
|
Re: Iterating over several lists at once
In article <1166017627.699257.166740@16g2000cwy.googlegroups.c om>,
"Gal Diskin" <gal.diskin@gmail.comwrote:
Quote:
Originally Posted by
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
>
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
>
What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...
|
Take a look at
http://mail.python.org/pipermail/py...ber/104983.html
or
http://aspn.activestate.com/ASPN/Co...n/Recipe/159975
There's nothing magic about either -- fundamentally, you're still doing an
N^3 operation and it's going to be slow. You might want to stop and think
if there's some better algorithm than an exhaustive search of the entire
domain space for whatever it is that you're trying to do.
Not the answer you were looking for? Post your question . . .
183,939 Experts ready to help you find a solution.
Sign up for a free account, or Login (if you're already a member).
|
