Transactional queue

(Cédric Krier) #1


Some actions may take a long time because they do a lot of things, for example processing a sale or a purchase. We have already separated the validation and the processing and added a cron task to process. But this does not scale up very well (one cron task at a time) and has a long delay (cron run every hours).
Solution like celery_tryton can not be used in core module because of the complexity of the dependencies and setup.


We need a way to post tasks in a queue and process them by workers as soon as possible. The queue must be transactional and scale well. The queue manager should be replaceable (ex: by Celery) like the Cache is.
For PostgreSQL back-end, we could use (or be inspired) by pq. For other backends like SQLite, we could just run the task directly as there are no LISTEN/NOTIFY support.

The posted task which should be a method on a Model should of course take parameters (*args, **kwargs) from the call but also store the user and context. The instantiate keyword will indicate which argument should be instantiate using the Model class.
A task will have no returned value because they are executed after the transaction (and the answer to the request).

The worker processes will wait for task to be posted in the queue and execute them one at a time. If an operational transaction happens, they will retry the task (same as in the dispatcher). For all other error, it will generate a log entry.


General low level:

Transaction().queue.put({'model': '', 'method': 'process', 'user': 1, 'args': [1], 'kwargs': {'context': {}}, 'instantiate': 0})

From Model:



(Sergi Almacellas Abellana) #2

Some doubts:

  • How will the workers run when celery is not installed?
  • Do we support scheduling tasks?
  • Do we support priority?
  • Do we manage retry on failure?

(Cédric Krier) #3

Like in pq using LISTEN/NOTIFY and advisory lock.

I think at one point the cron will just post the tasks in the queue.

Why not but it can come later.

Of course we have to retry on transactional error. But for other, we just log (or send an email to the user). The code who will be using such asynchronous task should implement a way to manually re-post the task.

(Sergi Almacellas Abellana) #4

I mean if workers will be exectued on a separated process, by cron or how. It’s not clear from the description.

(Cédric Krier) #5

Workers will be new processes.