`gevent.monkey` – Make the standard library cooperative#

Make the standard library cooperative.

The primary purpose of this module is to carefully patch, in place, portions of the standard library with gevent-friendly functions that behave in the same way as the original (at least as closely as possible).

The primary interface to this is the patch_all() function, which performs all the available patches. It accepts arguments to limit the patching to certain modules, but most programs should use the default values as they receive the most wide-spread testing, and some monkey patches have dependencies on others.

Patching should be done as early as possible in the lifecycle of the program. For example, the main module (the one that tests against __main__ or is otherwise the first imported) should begin with this code, ideally before any other imports:

from gevent import monkey
monkey.patch_all()

A corollary of the above is that patching should be done on the main thread and should be done while the program is single-threaded.

Tip

Some frameworks, such as gunicorn, handle monkey-patching for you. Check their documentation to be sure.

Warning

Patching too late can lead to unreliable behaviour (for example, some modules may still use blocking sockets) or even errors.

Tip

Be sure to read the documentation for each patch function to check for known incompatibilities.

Querying#

Sometimes it is helpful to know if objects have been monkey-patched, and in advanced cases even to have access to the original standard library functions. This module provides functions for that purpose.

is_module_patched()
is_object_patched()
get_original()

Plugins and Events#

Beginning in gevent 1.3, events are emitted during the monkey patching process. These events are delivered first to gevent.events subscribers, and then to setuptools entry points.

The following events are defined. They are listed in (roughly) the order that a call to patch_all() will emit them.

Each event class documents the corresponding setuptools entry point name. The entry points will be called with a single argument, the same instance of the class that was sent to the subscribers.

You can subscribe to the events to monitor the monkey-patching process and to manipulate it, for example by raising gevent.events.DoNotPatch.

You can also subscribe to the events to provide additional patching beyond what gevent distributes, either for additional standard library modules, or for third-party packages. The suggested time to do this patching is in the subscriber for gevent.events.GeventDidPatchBuiltinModulesEvent. For example, to automatically patch psycopg2 using psycogreen when the call to patch_all() is made, you could write code like this:

# mypackage.py
def patch_psycopg(event):
    from psycogreen.gevent import patch_psycopg
    patch_psycopg()

In your setup.py you would register it like this:

from setuptools import setup
setup(
    ...
    entry_points={
        'gevent.plugins.monkey.did_patch_builtins': [
            'psycopg2 = mypackage:patch_psycopg',
        ],
    },
    ...
)

For more complex patching, gevent provides a helper method that you can call to replace attributes of modules with attributes of your own modules. This function also takes care of emitting the appropriate events.

patch_module()

Use as a module#

Sometimes it is useful to run existing python scripts or modules that were not built to be gevent aware under gevent. To do so, this module can be run as the main module, passing the script and its arguments. For details, see the main() function.

Changed in version 1.3b1: Added support for plugins and began emitting will/did patch events.

get_original(mod_name, item_name)[source]#

Retrieve the original object from a module.

If the object has not been patched, then that object will still be retrieved.

Parameters:

mod_name (str) – The name of the standard library module, e.g., 'socket'. Can also be a sequence of standard library modules giving alternate names to try, e.g., ('thread', '_thread'); the first importable module will supply all item_name items.
item_name – A string or sequence of strings naming the attribute(s) on the module mod_name to return.

Returns:

The original value if a string was given for item_name or a sequence of original values if a sequence was passed.

is_module_patched(mod_name)[source]#

Check if a module has been replaced with a cooperative version.

Parameters:: mod_name (str) – The name of the standard library module, e.g., 'socket'.

is_object_patched(mod_name, item_name)[source]#

Check if an object in a module has been replaced with a cooperative version.

Parameters:

mod_name (str) – The name of the standard library module, e.g., 'socket'.
item_name (str) – The name of the attribute in the module, e.g., 'create_connection'.

main()[source]#

gevent.monkey - monkey patch the standard modules to use gevent.

USAGE: python -m gevent.monkey [MONKEY OPTIONS] [--module] (script|module) [SCRIPT OPTIONS]

If no MONKEY OPTIONS are present, monkey patches all the modules as if by calling patch_all(). You can exclude a module with –no-<module>, e.g. –no-thread. You can specify a module to patch with –<module>, e.g. –socket. In the latter case only the modules specified on the command line will be patched.

The default behavior is to execute the script passed as argument. If you wish to run a module instead, pass the --module argument before the module name.

Changed in version 1.3b1: The script argument can now be any argument that can be passed to runpy.run_path, just like the interpreter itself does, for example a package directory containing __main__.py. Previously it had to be the path to a .py source file.

Changed in version 1.5: The --module option has been added.

MONKEY OPTIONS: --verbose --[no-]socket, --[no-]dns, --[no-]time, --[no-]select, --[no-]thread, --[no-]os, --[no-]ssl, --[no-]subprocess, --[no-]sys, --[no-]builtins, --[no-]signal, --[no-]queue, --[no-]contextvars

patch_all(socket=True, dns=True, time=True, select=True, thread=True, os=True, ssl=True, subprocess=True, sys=False, aggressive=True, Event=True, builtins=True, signal=True, queue=True, contextvars=True, **kwargs)[source]#

Do all of the default monkey patching (calls every other applicable function in this module).

Returns:: A true value if patching all modules wasn’t cancelled, a false value if it was.

Changed in version 1.1: Issue a warning if this function is called multiple times with different arguments. The second and subsequent calls will only add more patches, they can never remove existing patches by setting an argument to False.

Changed in version 1.1: Issue a warning if this function is called with os=False and signal=True. This will cause SIGCHLD handlers to not be called. This may be an error in the future.

Changed in version 1.3a2: Event defaults to True.

Changed in version 1.3b1: Defined the return values.

Changed in version 1.3b1: Add **kwargs for the benefit of event subscribers. CAUTION: gevent may add and interpret additional arguments in the future, so it is suggested to use prefixes for kwarg values to be interpreted by plugins, for example, patch_all(mylib_futures=True).

Changed in version 1.3.5: Add queue, defaulting to True, for Python 3.7.

Changed in version 1.5: Remove the httplib argument. Previously, setting it raised a ValueError.

Changed in version 1.5a3: Add the contextvars argument.

Changed in version 1.5: Better handling of patching more than once.

patch_builtins()[source]#: Make the builtin __import__() function greenlet safe under Python 2.

Note

This does nothing under Python 3 as it is not necessary. Python 3 features improved import locks that are per-module, not global.

Deprecated since version 23.7.0: Does nothing on any supported platform.

patch_dns()[source]#

Replace DNS functions in socket with cooperative versions.

This is only useful if patch_socket() has been called and is done automatically by that method if requested.

patch_module(target_module, source_module, items=None)[source]#

Replace attributes in target_module with the attributes of the same name in source_module.

The source_module can provide some attributes to customize the process:

__implements__ is a list of attribute names to copy; if not present, the items keyword argument is mandatory. __implements__ must only have names from the standard library module in it.
_gevent_will_monkey_patch(target_module, items, warn, **kwargs)
_gevent_did_monkey_patch(target_module, items, warn, **kwargs) These two functions in the source_module are called if they exist, before and after copying attributes, respectively. The “will” function may modify items. The value of warn is a function that should be called with a single string argument to issue a warning to the user. If the “will” function raises gevent.events.DoNotPatch, no patching will be done. These functions are called before any event subscribers or plugins.

Parameters:: items (list) – A list of attribute names to replace. If not given, this will be taken from the source_module __implements__ attribute.
Returns:: A true value if patching was done, a false value if patching was canceled.

New in version 1.3b1.

patch_os()[source]#: Replace os.fork() with gevent.fork(), and, on POSIX, os.waitpid() with gevent.os.waitpid() (if the environment variable GEVENT_NOWAITPID is not defined). Does nothing if fork is not available.

Caution

This method must be used with patch_signal() to have proper SIGCHLD handling and thus correct results from waitpid. patch_all() calls both by default.

Caution

For SIGCHLD handling to work correctly, the event loop must run. The easiest way to help ensure this is to use patch_all().

patch_queue()[source]#: On Python 3.7 and above, replace queue.SimpleQueue (implemented in C) with its Python counterpart.

New in version 1.3.5.

patch_select(aggressive=True)[source]#

Replace select.select() with gevent.select.select() and select.poll() with gevent.select.poll (where available).

If aggressive is true (the default), also remove other blocking functions from select .

patch_signal()[source]#: Make the signal.signal() function work with a monkey-patched os.

Caution

This method must be used with patch_os() to have proper SIGCHLD handling. patch_all() calls both by default.

Caution

For proper SIGCHLD handling, you must yield to the event loop. Using patch_all() is the easiest way to ensure this.

See also

gevent.signal

patch_socket(dns=True, aggressive=True)[source]#

Replace the standard socket object with gevent’s cooperative sockets.

Parameters:: dns (bool) – When true (the default), also patch address resolution functions in socket. See Name Resolution (DNS) for details.

patch_ssl() → None[source]#

Replace ssl.SSLSocket object and socket wrapping functions in ssl with cooperative versions.

This is only useful if patch_socket() has been called.

patch_subprocess()[source]#: Replace subprocess.call(), subprocess.check_call(), subprocess.check_output() and subprocess.Popen with cooperative versions.

Note

On Windows under Python 3, the API support may not completely match the standard library.

patch_sys(stdin=True, stdout=True, stderr=True)[source]#

Patch sys.std[in,out,err] to use a cooperative IO via a threadpool.

This is relatively dangerous and can have unintended consequences such as hanging the process or misinterpreting control keys when input() and raw_input() are used. patch_all() does not call this function by default.

This method does nothing on Python 3. The Python 3 interpreter wants to flush the TextIOWrapper objects that make up stderr/stdout at shutdown time, but using a threadpool at that time leads to a hang.

Deprecated since version 23.7.0: Does nothing on any supported version.

patch_thread(threading=True, _threading_local=True, Event=True, logging=True, existing_locks=True) → None[source]#

Replace the standard thread module to make it greenlet-based.

Parameters:

threading (bool) – When True (the default), also patch threading.
_threading_local (bool) – When True (the default), also patch _threading_local.local.
logging (bool) – When True (the default), also patch locks taken if the logging module has been configured.
existing_locks (bool) – When True (the default), and the process is still single threaded, make sure that any threading.RLock (and, under Python 3, importlib._bootstrap._ModuleLock) instances that are currently locked can be properly unlocked. Important: This is a best-effort attempt and, on certain implementations, may not detect all locks. It is important to monkey-patch extremely early in the startup process. Setting this to False is not recommended, especially on Python 2.

Caution

Monkey-patching thread and using multiprocessing.Queue or concurrent.futures.ProcessPoolExecutor (which uses a Queue) will hang the process.

Monkey-patching with this function and using sub-interpreters (and advanced C-level API) and threads may be unstable on certain platforms.

Changed in version 1.1b1: Add logging and existing_locks params.

Changed in version 1.3a2: Event defaults to True.

patch_time()[source]#: Replace time.sleep() with gevent.sleep().

gevent.monkey – Make the standard library cooperative#

Querying#

Plugins and Events#

Use as a module#

`gevent.monkey` – Make the standard library cooperative#