Readthedocs is going to drop mercurial support

Organisation
1

Here is Start deprecating old VCS support · Issue #8840 · readthedocs/readthedocs.org · GitHub

So I think we should move away from readthedocs instead of trying to make a complex setup using for example our Github mirror.

I think to avoid being held hostages by service providers, it will be simpler to just build and host ourselves the documentation.
We could use a daily or weekly cron job and push the content for each series to our CDN.
Indeed we may use a single build from the top of the repository of tryton.
For other projects we just need to build them and push to CDN.

1 Like
2

One small benefit of moving out from readthedocs is to no more have Odoo advertise next to our documentation.

3

Also they send us an email about the change:

Hello,
You are receving this email because your Read the Docs project is impacted by an upcoming deprecation.

We are discussing dropping of support for Bazaar, Mercurial and Subversion version system controls. This means we will only support Git to do the initial clone and build your project’s documentation. We would like your feedback on this change, to better understand if you can use Git for your project.

We’ve made this decision because 99% of our users use Git and we can’t cover the maintainance cost we were spending on Bazaar, Mercurial and SVN. Over time, these VCS tools have not gotten new features, and add confusion to users because not all our features work for them.

Unfortunately, there is no workaround on the Read the Docs side that you can follow to keep building your documentation using these VCSs, but you could probably import your Subversion or Mercurial repository into GitHub or similar services to keep building on Read the Docs.

The projects that are impacted by this change are:

Please contact us or reply to this email if you have any questions, or think there is a strong reason we should continue to offer basic support for other VCS tools.

Thanks, Read the Docs
Keep documenting,
Read the Docs

But for me reading the issue, they already have taken their decision and even if we would succeed to convince them to keep mercurial. It will just be a matter of time before they start again to want to remove mercurial because it is clear that they do not want to use abstraction over the VCS but only Git.

4

I think the availability of the documentation is a crucial pin point for the successful onboarding in Tryton. That’s why I asked already in How to build documentation for a way to build the available documentation in order to package it.

Because we built with tryton-server-all-in-one (s. Debian integration packages for Tryton) a highly integrated deployment with a production ready database, but when it comes to select from the available modules for activation the user is severly lacking information about the purpose of each module. It would be a great step forward to make the documentation directly available.

1 Like
5

Another point is that if we have a global documentation build we can easily include a section for transversal documentation (so we could remove the Howto - Tryton Discussion category and have a MR workflow for them).

1 Like
6

Indeed we can almost use .gitlab-scripts/generate-doc.sh as-is to generate the documentation website.
We just need to update readthedocs folder to use proper path (probably relative).

One missing feature is that the search from the top site will not include result from “subprojects” like it does with rtd. I found sphinx-affiliates extension for this purpose but it is no more maintained and the author does not know what to do with it.

Another option would be do have a single build but i think it is important to be able to build the documentation of just one module (and have links using sphinx.ext.intersphinx).

7

I looked how sphinx implement the search. It is based on 2 files _static/searchtools.js which contains the engine and searchindex.js which contains the data.
The searchindex.js is loaded by calling Search.setIndex. The structure of the file is an object with keys:

  • docnames: a list of document names
  • filenames: a list of corresponding names
  • titles: a list of corresponding titles
  • terms: an object with term as attribute and a list of index in the previous lists.
  • …

So we could merge all the indexes of the sphinx documentations as long as we prefix the docname with the folder name of the package.

I made a simple test by editing the file and it is working.

Now should the merge be done in the browser or as post-process of the build?

8

If we host ourselves the documentation, Remove translation in readthedocs URL (#12833) · Issues · Tryton / Tryton · GitLab will be no more relevant.

9

I filled Remove readthedocs (!1245) · Merge requests · Tryton / Tryton · GitLab which allow to reuse .gitlab-scripts/generate-doc.sh for hosting.
I have setup the server to build and serve the result. It can be browsed from documentation-10f05.kxcdn.com but some links (created by intersphinx) are still absolute so they point to https://docs.tryton.org/ but at the end this hostname will be pointing to the CDN.

The only missing point is the global searchindex but this can be done on a second time.

The next challenge will be to backport all the changes to supported series as they have less modules.

10

The migration has been done. Now https://docs.tryton.org/ is served by ourselves. The build is done once every night.
Relatorio and Goocalendar has also been moved to Relatorio — A templating library able to output odt and pdf files and goocalendar — Calendar widget using GooCanvas — A calendar widget for GTK using PyGoocanvas.

I started to delete readthedocs projects to avoid people accessing the old versions directly but it takes times as it must be done one by one.

This is still missing:

4 Likes
11

I implemented the merge of the indexes.
Here is the script.

merge_searchindex.py 
#!/usr/bin/env python

import json
import os
import shutil
import sys
import copy
import glob

PREFIX = 'Search.setIndex('
SUFFIX = ')'


def main(dirname):
    indexes = {}
    index_files = glob.glob(
        os.path.join(dirname, '**', 'searchindex.js'), recursive=True)
    for index_file in index_files:
        index_file_orig = f'{index_file}.orig'
        if os.path.exists(index_file_orig):
            to_load = index_file_orig
        else:
            shutil.copyfile(index_file, index_file_orig)
            to_load = index_file
        indexes[index_file] = load_index(to_load)

    for target_name, target in indexes.items():
        target = copy.deepcopy(target)
        for index_file, index in indexes.items():
            if index_file != target_name:
                prefix = os.path.relpath(
                    os.path.dirname(index_file),
                    os.path.dirname(target_name))
                merge_index(target, index, prefix)
        with open(target_name, 'w') as fp:
            fp.write(PREFIX)
            fp.write(json.dumps(target))
            fp.write(SUFFIX)


def load_index(filename):
    with open(filename, 'r') as fp:
        content = fp.read()
        return json.loads(content[len(PREFIX):-len(SUFFIX)])


def merge_index(target, index, prefix):
    file_offset = len(target['filenames'])
    obj_offset = len(target['objnames'])

    for filename in index['filenames']:
        target['filenames'].append(os.path.join(prefix, filename))
    for docname in index['docnames']:
        target['docnames'].append(os.path.join(prefix, docname))
    for title in index['titles']:
        target['titles'].append(title)

    for title, titles in index['alltitles'].items():
        for file, id in titles:
            target['alltitles'].setdefault(title, []).append(
                [file + file_offset, id])

    for entry, entries in index['indexentries'].items():
        try:
            for file, id, is_main in entries:
                target['indexentries'].setdefault(entry, []).append(
                    [file + file_offset, id, is_main])
        except ValueError:
            for file, id in entries:
                target['indexentries'].setdefault(entry, []).append(
                    [file + file_offset, id])

    for key, value in index['objnames'].items():
        target['objnames'][str(int(key) + obj_offset)] = value

    for key, value in index['objtypes'].items():
        target['objtypes'][str(int(key) + obj_offset)] = value

    for object_, objects in index['objects'].items():
        for file, obj, score, anchor, name in objects:
            target['objects'].setdefault(object_, []).append(
                [file + file_offset, obj + obj_offset, score, anchor, name])

    for word, files in index['terms'].items():
        if not isinstance(files, list):
            files = [files]
        if not isinstance(target['terms'].setdefault(word, []), list):
            target['terms'][word] = [target['terms'][word]]
        for file in files:
            target['terms'][word].append(file + file_offset)

    for word, files in index['titleterms'].items():
        if not isinstance(files, list):
            files = [files]
        if not isinstance(target['titleterms'].setdefault(word, []), list):
            target['titleterms'][word] = [target['titleterms'][word]]
        for file in files:
            target['titleterms'][word].append(file + file_offset)


if __name__ == '__main__':
    try:
        dirname = sys.argv[1]
    except IndexError:
        dirname = '.'
    main(dirname)

It is hosted on the server and backup like the others.

It should be available in the next days after nightly rebuild and CDN refresh.

I also created Use base URL as logo link and include homepage in TOC (!2203) · Merge requests · Tryton / Tryton · GitLab to unify the navigation.

1 Like