The synchronous code in asynchronous, Twisted, or tale of how to cross a hedgehog with a snake
All is well
Twisted — asynchronous (event-driven) framework written in Python. A powerful tool for rapid development of the network (and not only) services. It is designed with pattern design Reactor. Services created using Twisted fast and reliable framework allows you to write spaghetti code, full of strange kavakami, has within itself a beautiful helper (Deferred, Transport, Protocol etc). In other words, makes our life backend developers better.
But
The main problem is that numerous, reliable, tested, convenient library that is at its core a synchronous Python modules (socket, os, ssl, time, select, thread, subprocess, sys, signal, etc), just take and will block us the process, a loop reactor and there will be trouble. Such libraries, for example, are psycopg2, request, mysql and others. In particular, psycopg2 is used in the Django ORM as one of the backend databases.
What do you do?
There are three ways. Difficult, acceptable, and good. Difficult to realize the analog library in Twisted. Acceptable — use deferToThread to run synchronous code in separate threads (using a thread pool implemented in Twisted). On a good way (in my opinion) and will be discussed in a note.
Use "green" threads, and events for context switching!
What do we need?
the
-
the
- Greenlets are lightweight "green" threads, which run inside the main application process the
- Gevent framework that allows you to switch context between greenlease, in that moment, when the executable code is locked the
- Method of the reactor [deferToGreenlet], allowing wrap to the Deferred greenlet
Example of the use of technology in a real project
I didn't want to write your own implementation of the reactor with the ability to send code to greenlets are, as you've found the solution, tested and implemented in the project. Code reactor can be collected here.
To use geventreactor during the application initialization, need to install it:
the
from geventreactor import install
install()
Now we have available new methods:
the
__all__ = ['deferToGreenletPool', 'deferToGreenlet', 'callMultipleInGreenlet', 'waitForGreenlet', 'waitForDeferred',
'blockingCallFromGreenlet', 'IReactorGreenlets', 'GeventResolver', 'GeventReactor', 'install']
By analogy with the reactor.deferToThread(f, *args, **kwargs), you can call reactor.deferToGreenlet(f, *args, **kwargs), where f — callable object, and *args and **kwargs arguments.
To work you also need to patch the library in the namespace:
the
from gevent import monkey
monkey.patch_all()
After these manipulations, the basic Python libraries will be patched by Gevent. See the documentation for Gevent
Now all libraries or code that directly imports them, if you call blocking methods or functions will cause the appropriate event Gevent. At these events hanging colbecki that allows you to switch context between greenlite.
My project uses Django ORM to manipulate the data in PostgreSQL. Therefore, in order to ORM methods does not block the process need to use a special backend that allows you to create a pool of database connections and switch between connections. One of the backends is django-db-geventpool
To use django-db-geventpool not difficult. Just follow the documentation.
What?
Method reactor.deferToGreenlet returns a Deferred object, which can be run as normal Deferred.
For example, we have a model:
the
class ExampleModel(models.Model):
title = models.CharField(max_length=256)
We want to get all the models and pass them to some kind of processor inside the system. We can write something like:
And our code will not block the main process. Because in that moment, when the Django ORM will cause cursor.execute () which will wait for a response from the database driver geventreactor will switch the context to another Deferred.
What?
We can perform the synchronous code within Twisted, without creating extra threads or processes, while not blocking the event loop of the reactor. The main thing is to follow the basic principles of asynchronous systems, pieces of code should not be run too long, gevent allows you to switch context from any place of code where it is convenient to us, just call gevent.the sleep().
Comments
Post a Comment