More flexibility managing worker queues

Feature Ideas
1

We’ve got a case where we want one of the queues to be executed in a specific dedicated server.

That is possible because trytond-worker allows adding a worker for a single queue by specifying the --name parameter.

Then, we want another worker to manage the rest of the queues, but that is not possible: we must create a worker for each of the queues.

For that, it seems like a parameter such as “–except-name” would be useful.

It would also be useful that both “–name” and “–except-name” would accept several queue names.

Opinions?

1 Like
2

Why you need such feature?
From my understanding as far a worker is free it makes sense to execute any process on it.

3

One of the queues (the one in the dedicated server) has a lot of tasks. If all resources are dedicated to this queue, then other tasks can take a long time to be processed, which is not desired.

That’s exactly the reason why tryton already supports several queue names.

1 Like
4

In this case I will use two workers:

  • One with the named queue filter and as much as possible worker threads
  • Another without name filter with less worker threads.

This will ensure that queue with a lot of tasks is processed fast while having some resources to others. You can run the first on a dedicated hardware and the other on shared hardware if you prefer.

Have you tried such setup?

5

The setup you suggest does not work because the second worker you propose will also pick tasks from the named queue.

Let’s put an example so it’s easier to explain:

  • Tasks are added to queues named A, B, and C
  • Queue A has usually thousands of tasks
  • Queue B and C have much fewer tasks but should not be delayed too much if there are lots of tasks on A, which would take hours to process

If you setup two queues as you proposed it would look something like this:

a) trytond-worker -n 8 --name A
b) trytond-worker -n 2

But this will cause worker a) to pick tasks from queue A and b) to pick tasks from queues A, B or C.

This means that tasks in B or C queues may be delayed several hours.

We could solve that with:

a) trytond-worker -n 8 --name A
b) trytond-worker -n 1 --name B
c) trytond-worker -n 1 --name C

but this means that if more queues are used in the future, we must add new workers.

But what we really need should be easier:

a) trytond-worker -n 8 --name A
b) trytond-worker -n 2 --except-name A

Hope its clearer now.

1 Like
7

Of course but only if other named workers does not have enought power to finish al pending tasks.

For me we should keep this behaviour to reduce the number of pending tasks and prevent having spare workers.

If you want a queue to not be delayed just create a worker for it. It’s the way to give priority for it.

When more queues are added you have two options:

  • If you want to priorize the queue just add a worker for it
  • If you do not want, just leave it on the default worker.

With you proposal (using except) adding new workers may lower the priority of tasks that should not be delayed to much, so it may produce hidden problems.

With current behaviour, we explicity define which tasks we want to priorize and always have a fallback for the rest.

Please take care that when worker is enabled you lose the control about when tasks are finished. They will be finished but you do not know when.

If you have a tasks which takes a lot of time, i think we should focus on improving it’s performance.

8

Such feature was already discussed here.

Having many trytond-worker services does not work as expected as described @albert.
We tried to get several services for dedicated queues and finally came back to a simple deployment with an unique worker service an many processes.