Yesterday and today i get from time to time a gateway time out 504 on translate.tryton.org .
After a while a can reconnect.
Anny one having the same problem?
kind regards
Bert
The 4 µwsgi processes were fully busy and nothing in the logs.
I restarted the server and now it seems normal.
Side note: this week, I updated the server with a new kernel so there was a reboot. And I have already seen that the auto-restart of docker may sometime cause trouble on reboot.
thanx for the info.
works fine now.
Once again on saterday 20/07/2019 21:43 failed to load machine translations, and after a while when refreshing 504 ,
I have restart it again. It looks that for some requests (I suspect crawlers) weblate goes into an infinite loop. Unfortunately the harakiri of µwsgi is set to 1 h.
I do not want to spend more time investigate the Django applications because they are too complex and bloated for me.
So if it happens again, just ping me on IRC. I would restart it.
I have activated PythonTracebacker on the server. So next time, I should get more information about those never ending requests.
The problem comes from the translation memory.
Here is the traceback of a staling worker:
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/raven/middleware.py lineno = 100 function = __call__ line = iterable = self.application(environ, start_response)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/wsgi.py lineno = 141 function = __call__ line = response = self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py lineno = 75 function = get_response line = response = self._middleware_chain(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/weblate/middleware.py lineno = 58 function = __call__ line = return self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py lineno = 94 function = __call__ line = response = response or self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py lineno = 94 function = __call__ line = response = response or self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py lineno = 94 function = __call__ line = response = response or self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py lineno = 94 function = __call__ line = response = response or self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py lineno = 94 function = __call__ line = response = response or self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/weblate/accounts/middleware.py lineno = 58 function = __call__ line = return self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py lineno = 94 function = __call__ line = response = response or self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py lineno = 94 function = __call__ line = response = response or self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py lineno = 94 function = __call__ line = response = response or self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/weblate/accounts/middleware.py lineno = 141 function = __call__ line = return self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/weblate/middleware.py lineno = 67 function = __call__ line = response = self.get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py lineno = 34 function = inner line = response = get_response(request)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py lineno = 113 function = _get_response line = response = wrapped_callback(request, *callback_args, **callback_kwargs)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/weblate/trans/views/js.py lineno = 68 function = translate line = request
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/weblate/machinery/base.py lineno = 298 function = translate line = source, language, text, unit, request
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/weblate/machinery/weblatetm.py lineno = 57 function = download_translations line = ).more_like_this(unit, 1000)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/weblate/trans/models/unit.py lineno = 249 function = more_like_this line = more_results = Fulltext().more_like(unit.pk, unit.source, top)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/weblate/trans/search.py lineno = 216 function = more_like line = (h['pk'], h.score) for h in searcher.search(query, limit=top)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/whoosh/searching.py lineno = 786 function = search line = self.search_with_collector(q, c)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/whoosh/searching.py lineno = 819 function = search_with_collector line = collector.run()
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/whoosh/collectors.py lineno = 144 function = run line = self.collect_matches()
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/whoosh/collectors.py lineno = 214 function = collect_matches line = for sub_docnum in self.matches():
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/whoosh/collectors.py lineno = 409 function = matches line = self.skipped_times += matcher.skip_to_quality(minscore)
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/whoosh/matching/binary.py lineno = 289 function = skip_to_quality line = while a.is_active() and b.is_active() and aq + bq <= minquality:
thread_id = uWSGIWorker3Core0 filename = /usr/local/lib/python3.7/dist-packages/whoosh/matching/wrappers.py lineno = 74 function = is_active line = return self.child.is_active()
According to https://github.com/WeblateOrg/weblate/commit/a860bb42533035b9ca2ace06015a5c3eb39c8803 such performance issue should not happen with up to date indexes. So I manually started a rebuild of the index: /usr/local/bin/weblate rebuild_index tryton.