Server graceful shutdown

Hello Everyone,

I was wandering if upon shutdown, the server performs a graceful shutdown? For example, I have trytond running as a systemd service. It is configured as a target, so that when main server is started, it starts the cron and workers. So what if the cron is in the middle of a scheduled task and we issue a “stop” to trytond’s systemd service, would the cron server stops abruptly or will it finish processing the task before stopping?

I have a suspicion that it stops abruptly, because I noticed some very rare unexpected results happening in my scheduled tasks, and I can’t find why, but I suspect they may be happening when the server restarts.

The services are just stopping at the moment they receive the signal. But it is not a problem as everything is under a SQL transaction so the started transaction are just rollbacked and only the state before them is stored in the database.
The only possible problem is if you have code that is not transactional (but there should be any in standard module).

Yes, it is not transactional, because it has to do with sending data to an external server that I have no control, so cannot implement any transaction-like protection.

So if I were to fix this problem is there is a way to modify trytond-cron server to catch the signal and shutdown only after it is safe? Could this become a feature for Trytond in the future?

Well it should probably be possible to implement a DataManager even if it has to be a sidekick service that will deal with the external server.

I doubt and anyway this will not make it transactional as process can be forced to stop anyway.

I understand it will not make it transactional and hence it is not a perfect solution, but at least it reduces the likelyhood of errors on a graceful shutdown. I see this feature as implemented on other servers like PostgreSQL where the server doesn’t end immediately, but just stops receiving new connections and finishes all sessions before terminating.
See: PostgreSQL: Documentation: 9.1: Shutting Down the Server

For trytond-cron, upon receiving the SIGTERM, it will just stops processing the next task and finishes the current task.

Not convinced you can get a good solution if you mix transactional/not-transactional activities.
When I stop a server, it is often because I want to be sure all sessions are cleaned , even those which never end. An example is to stop any access to the database in order to drop it.

There are situations which are not as transaction as one would wish, say for example I have a schedule task that uploads electronic invoices to the government tax department. I can transactionaly mark each invoice that was uploaded successfully and if the server is terminated abruptly during an upload, the DB will rollback and this last invoice will not be flagged as successful. However, the invoice was actually uploaded completely, hence upon starting trytond-cron it will upload again which is wrong and probably troublesome as it may be seen as a duplicate by the government.
Same thing could happen if you have a task that emails customer information which was interrupted just right at the end before it could update the status to the DB, hence the email would be send out again creating confusion.

The feature that I ask to implement to the trytond-cron for a graceful shutdown is not really that difficult and would be harmless and even unnoticeable. It just catches the SIGTERM signal and stops after the threads are join.

I have created a feature request Issue 10793: Add graceful shutdown for trytond-cron - Tryton issue tracker
I will upload my code to the review code site.

The two examples are not really good examples. For the email case, we have the SMTPDataManager that handle sending email following the transaction. And for the external upload, if the external service does not provide idempotent way or a check to prevent duplicate upload, no external design can make it work correctly.

I don’t know why is it so bad to have the cron server terminates after ensuring all its threads have return. It doesn’t add any overhead nor extra memory to do this. Here’s the code review issue375551002 (Issue 375551002: Graceful shutdown trytond-cron - Code Review)

Because for me it gives the false impression that it is safe to not be transactional.