gevent.pool – Managing greenlets in a group

The Group class in this module abstracts a group of running greenlets. When a greenlet dies, it’s automatically removed from the group. All running greenlets in a group can be waited on with Group.joinall(), or all running greenlets can be killed with Group.kill().

The Pool class, which is a subclass of Group, provides a way to limit concurrency: its spawn method blocks if the number of greenlets in the pool has already reached the limit, until there is a free slot.

class Group(*args)

Bases: gevent.pool.GroupMappingMixin

Maintain a group of greenlets that are still running.

Links to each item and removes it upon notification.

add(greenlet)
apply(func, args=None, kwds=None)

Rough quivalent of the apply() builtin function blocking until the result is ready and returning it.

The func will usually, but not always, be run in a way that allows the current greenlet to switch out (for example, in a new greenlet or thread, depending on implementation). But if the current greenlet or thread is already one that was spawned by this pool, the pool may choose to immediately run the func synchronously.

apply_async(func, args=None, kwds=None, callback=None)

A variant of the apply() method which returns a Greenlet object.

If callback is specified then it should be a callable which accepts a single argument. When the result becomes ready callback is applied to it (unless the call failed).

apply_cb(func, args=None, kwds=None, callback=None)
discard(greenlet)
full()
greenlet_class

alias of Greenlet

imap(func, *iterables, maxsize=None) → iterable

An equivalent of itertools.imap(), operating in parallel. The func is applied to each element yielded from each iterable in iterables in turn, collecting the result.

If this object has a bound on the number of active greenlets it can contain (such as Pool), then at most that number of tasks will operate in parallel.

Parameters:maxsize (int) –

If given and not-None, specifies the maximum number of finished results that will be allowed to accumulate awaiting the reader; more than that number of results will cause map function greenlets to begin to block. This is most useful is there is a great disparity in the speed of the mapping code and the consumer and the results consume a great deal of resources.

Note

This is separate from any bound on the number of active parallel tasks, though they may have some interaction (for example, limiting the number of parallel tasks to the smallest bound).

Note

Using a bound is slightly more computationally expensive than not using a bound.

Tip

The imap_unordered() method makes much better use of this parameter. Some additional, unspecified, number of objects may be required to be kept in memory to maintain order by this function.

Returns:An iterable object.

Changed in version 1.1b3: Added the maxsize keyword parameter.

imap_unordered(func, *iterables, maxsize=None) → iterable

The same as imap() except that the ordering of the results from the returned iterator should be considered in arbitrary order.

This is lighter weight than imap() and should be preferred if order doesn’t matter.

See also

imap() for more details.

join(timeout=None, raise_error=False)
kill(exception=<class 'greenlet.GreenletExit'>, block=True, timeout=None)
killone(greenlet, exception=<class 'greenlet.GreenletExit'>, block=True, timeout=None)
map(func, iterable)

Return a list made by applying the func to each element of the iterable.

See also

imap()

map_async(func, iterable, callback=None)

A variant of the map() method which returns a Greenlet object that is executing the map function.

If callback is specified then it should be a callable which accepts a single argument.

map_cb(func, iterable, callback=None)
spawn(*args, **kwargs)

Begin a new greenlet with the given arguments (which are passed to the greenlet constructor) and add it to the collection of greenlets this group is monitoring.

Returns:The newly started greenlet.
start(greenlet)

Start the un-started greenlet and add it to the collection of greenlets this group is monitoring.

wait_available()
class Pool(size=None, greenlet_class=None)

Bases: gevent.pool.Group

Create a new pool.

A pool is like a group, but the maximum number of members is governed by the size parameter.

Parameters:size (int) –

If given, this non-negative integer is the maximum count of active greenlets that will be allowed in this pool. A few values have special significance:

  • None (the default) places no limit on the number of greenlets. This is useful when you need to track, but not limit, greenlets, as with gevent.pywsgi.WSGIServer
  • 0 creates a pool that can never have any active greenlets. Attempting to spawn in this pool will block forever. This is only useful if an application uses wait_available() with a timeout and checks free_count() before attempting to spawn.
add(greenlet)
apply(func, args=None, kwds=None)

Rough quivalent of the apply() builtin function blocking until the result is ready and returning it.

The func will usually, but not always, be run in a way that allows the current greenlet to switch out (for example, in a new greenlet or thread, depending on implementation). But if the current greenlet or thread is already one that was spawned by this pool, the pool may choose to immediately run the func synchronously.

apply_async(func, args=None, kwds=None, callback=None)

A variant of the apply() method which returns a Greenlet object.

If callback is specified then it should be a callable which accepts a single argument. When the result becomes ready callback is applied to it (unless the call failed).

apply_cb(func, args=None, kwds=None, callback=None)
discard(greenlet)
free_count()

Return a number indicating approximately how many more members can be added to this pool.

full()

Return a boolean indicating whether this pool has any room for members. (True if it does, False if it doesn’t.)

greenlet_class

alias of Greenlet

imap(func, *iterables, maxsize=None) → iterable

An equivalent of itertools.imap(), operating in parallel. The func is applied to each element yielded from each iterable in iterables in turn, collecting the result.

If this object has a bound on the number of active greenlets it can contain (such as Pool), then at most that number of tasks will operate in parallel.

Parameters:maxsize (int) –

If given and not-None, specifies the maximum number of finished results that will be allowed to accumulate awaiting the reader; more than that number of results will cause map function greenlets to begin to block. This is most useful is there is a great disparity in the speed of the mapping code and the consumer and the results consume a great deal of resources.

Note

This is separate from any bound on the number of active parallel tasks, though they may have some interaction (for example, limiting the number of parallel tasks to the smallest bound).

Note

Using a bound is slightly more computationally expensive than not using a bound.

Tip

The imap_unordered() method makes much better use of this parameter. Some additional, unspecified, number of objects may be required to be kept in memory to maintain order by this function.

Returns:An iterable object.

Changed in version 1.1b3: Added the maxsize keyword parameter.

imap_unordered(func, *iterables, maxsize=None) → iterable

The same as imap() except that the ordering of the results from the returned iterator should be considered in arbitrary order.

This is lighter weight than imap() and should be preferred if order doesn’t matter.

See also

imap() for more details.

join(timeout=None, raise_error=False)
kill(exception=<class 'greenlet.GreenletExit'>, block=True, timeout=None)
killone(greenlet, exception=<class 'greenlet.GreenletExit'>, block=True, timeout=None)
map(func, iterable)

Return a list made by applying the func to each element of the iterable.

See also

imap()

map_async(func, iterable, callback=None)

A variant of the map() method which returns a Greenlet object that is executing the map function.

If callback is specified then it should be a callable which accepts a single argument.

map_cb(func, iterable, callback=None)
spawn(*args, **kwargs)

Begin a new greenlet with the given arguments (which are passed to the greenlet constructor) and add it to the collection of greenlets this group is monitoring.

Returns:The newly started greenlet.
start(greenlet)

Start the un-started greenlet and add it to the collection of greenlets this group is monitoring.

wait_available(timeout=None)

Wait until it’s possible to spawn a greenlet in this pool.

Parameters:timeout (float) – If given, only wait the specified number of seconds.

Warning

If the pool was initialized with a size of 0, this method will block forever unless a timeout is given.

Returns:A number indicating how many new greenlets can be put into the pool without blocking.

Changed in version 1.1a3: Added the timeout parameter.

Next page: gevent.queue – Synchronized queues