[Tutor] Class-based generator

Oscar Benjamin oscar.j.benjamin at gmail.com
Mon Feb 18 09:58:35 CET 2013

On 18 February 2013 07:36, Michael O'Leary <michael at seomoz.org> wrote:
> I wrote some code to create tasks to be run in a queue based system last
> week. It consisted of a big monolithic function that consisted of two parts:
> 1) read data from a file and create dictionaries and lists to iterate
> through
> 2) iterate through the lists creating a job data file and a task for the
> queue one at a time until all of the data is dealt with
> My boss reviewed my code and said that it would be more reusable and
> Pythonic if I refactored it as a generator that created job data files and
> iterated by calling the generator and putting a task on the queue for each
> job data file that was obtained.
> This made sense to me, and since the code does a bunch of conversion of the
> data in the input file(s) to make it easier and faster to iterate through
> the data, I decided to create a class for the generator and put that
> conversion code into its __init__ function. So the class looked like this:

It's not a "generator" if you create a class for it. Your class is
(trying to be) an iterator.

> class JobFileGenerator:
>     def __init__(self, filedata, output_file_prefix, job_size):
>         <convert filedata to a more usable form>
>     def next(self):
>         while <there is more data>:
>             <yield a job data file>

next() should return a single item not a generator that yields items.
If you perhaps rename the next function as __iter__ then it will be a
proper iterator.

I suspect however that your boss just wants you to write a single
generator function rather than an iterator class. For example:

def generate_jobs():
    while <data>:
        yield <job>

for job in generate_jobs():
    <queue job>

More information about the Tutor mailing list