Migration of development to Heptapod

Organisation
1

Our current development infrastructure is aging and become more complicated to maintain.
Rietveld is mainly no more maintained. Our drone fork is stuck in version 0.8 when upstream is at 2.91. Roundup is not really good for searching.
There have been already many discussion about this:

Our idea is to migrate the development to Heptapod, a derived version of GitLab that supports Mercurial.
So we asked to Octobus how we could plan such migration.

Hosting

Octobus tells us that we are eligible to be hosted freely on https://foss.heptapod.net/. But if we are going to be hosted there, we proposed to make an annual contribution in the form of support contract by the Foundation.

Repository

In order to benefit from the full GitLab feature, it is required that we merge the Tryton’s repository into a mono-repository. We have already discussed such possibility on Using a mono-repository - #5 by pokoli.
Globally the idea is to keep the directory structure we have with tryton-env repository (to ease partial clone with ConvertExtension).
There are still some points that need to be clarified but Octobus has experiences with such tasks:

Rewrite history?
Tag naming convention
Clean content (remove .drone.yml, etc.)
Conversion script

Workflow

Heptapod workflow is based on merge request. We think we should keep the strategy to have a linear history on merge.
We have identified one issue with the current workflow which is that there is no fork but a single repository with topics. So this force new user to request access to the repository in order to make a merge request. But this could be solved with the implementation of user namespaces for topics. We have a quotation from Octobus to implement this feature.

Configuration workflow
User namespaces for topics

Tests

The tests will be run by the CI/CD of Heptapod. According to Octobus the load of our tests should not be a problem for their hosting. But if necessary we could provide extra hosts to run some jobs.

Indeed the main difficulty with the tests is the management of the installation of locale dependencies when running inside tox the tests of one module (that depends on other). For now we are relying on https://trydevpi.tryton.org/ to find and clone other modules. But it seems we could fill a directory with archives of every modules (probably just sdist) and use it as --find-links option of pip.

The second one is that the jobs run the tests only for modified modules. This could probably be done with a script that retrieve the list of modules from the mercurial output.
We may need an option (for core devs) to trigger the tests for all modules.

In addition to tests we need also jobs to run flake8 and isort on the modified files. And we could also add a weekly job to run tests for all modules (last Python version and sqlite only).

Dependency installation
Pipeline configuration

  • Tests
  • Linters

Bug tracker

The issues need to be imported in Heptapod. Normally we should be able to keep the same ID for the issues (which may allow to rewrite the commit messages to include link to the migrated issues).
The attachment on the issue have also to be imported.
Roundup user will be automatically registered as Heptapod users.

Data import

Documentation

GitLab can be integrated with readthedocs but as our documentation is divided per module. The generic plugin will probably need to be customized. Also we need that new module can be done from the configuration UI of Heptapod and readthedocs.

Setup web-hook

Tools

The scripts used to make a release must be adapted to use the mono-repository design.

release scripts

Other side projects

The other projects like relatorio, python-sql, chronos and goocalendar will be migrated to Heptapod as sub-projects. The same migration for issues will be performed and the repository will be just pushed as-is.

Implementation

We have a quotation that we will submit to the Foundation Board from Heptapod for the tasks:

  • Repository migration
  • Data import from roundup
  • CI/CD configuration
  • topics namespaces

I expect to have to spend about 3-4 days to follow the process and provide information with Octobus.

8 Likes
3

Is also possible to update the links of the bugtracker and codereview in the different posts here on discuss?

1 Like
4

For the link to the bugtracker we could setup a redirection.
For the reviews, I think we can not do much except keeping the instance running as readonly for a while.

5

I was more thinking about replacing the links here on discuss with the new url for the bugreport or the commithash for the codereview.

Otherwise, please add a redirect for both and maybe edit the relevant posts here on discuss with a small message saying something like We have moved to ............ so the links above won't work anymore. So people know.

6

The Foundation agreed on spending the cost to support this migration.
Now we are just waiting more feed backs from the community to start the process.

6 Likes
7

It sounds like a good plan, and if it makes things easier to maintain I think it will be a good improvement. :+1:

8

Change is always complicated but Tryton needs this change and Heptapod seems the best solution.

9

In principle I welcome very much the move to a standard and up-to-date platform.

Open questions worth to be evaluated/answered:

Since it is a major effort to migrate to a new platform and major changes to the repository structure have to be done anyway I would be interested what still is the major benefit of staying with mercurial vs. git. In the interests to migrate to a best possible solution in terms of durabililty and sustainability here some observations from my part:

With heptapod we would be using a layer over original gitlab that could bring some complications.

aszs
How do you manage your fork vis a vis upstream gitlab? It seems challenging considering how complicated gitlab is.

Hi ! Heptapod maintainer here. Logilab is one of our users and a sponsor, indeed. Yes technically it is a serious effort

  • the very same company octobus is also responsible of a substantial part of the maintenance of mercurial itself (commit history of mercurial)

So while there is no doubt about the skills of the heptapod developers for me there is still the now even more important question if it is not finally the time to rely on a more future proof stack: why not use the original (gitlab) if the effort has to be done anyway?

1 Like
10

We have been reassured by our talk with Octobus that indeed this layer is very thin and limited to a small part of Gitlab. So it is mainly Gitlab that will be run.

But as good Open Source citizen the Foundation will take his responsibility and support the provider financially to ensure a part of his sustainability.
Similar though can be applied to Tryton so…

Have you a proposal with a quotation to propose for such migration? (as it has been asked for years to anyone who make such request).

Finally changing from mercurial to git will be an even bigger effort. We will have to rewrite all the existing tools. And the majority of core developers do not want to switch to Git and prefer the clean and simplicity of mercurial.

1 Like
11

Re. possible history rewrite and Conversion script: I did some repository conversions using reposurgeon. The IMHO most important feature is that conversion can be scripted and thus easily tested and adjusted.

When I used it some years ago, it was written in Python (but an ugly piece of code), meanwhile it has been rewritten in golang :frowning:

12

…and gitlab as well of course.

My question really doesn’t go into the direction of preferences, but pure future proofness of the chosen tool. Gathering statistics covers this side of the problem.

For all I see mercurial is actually much less used and - the more important fact - constantly decreasing over the last years. May be we will see this discussion in few years again…

13

If software choice was really a popularity contest then nobody should be here.
I have no doubt in the future of Mercurial. Its development is still going on, they pushed new concepts like phase, topics and rust implementation over the last years. It used by big players.

1 Like
14

While I am a big fan of mono-reposotories, and I think they are worth the effort regardless of an Heptapod/whatever migration, I don’t quite undrestand why this is a requirement. But I’ll celebrate this.

I think the point is: if Tryton considers moving from development platform, by replacing Mercurial with Git it would be possible to take advantatge of even more platforms, which could lead to other quotations if this was something that could be discussed.

A Mercurial to Git conversion is pretty straightforward, so that a final quotation would be pretty similar the one under discussion. Probably such a quotation has never been presented due to the upfront confrontation this idea recieves, and not due to viability reasons.

I am unaware of the extent of the tooling, but I can imagine that dropping DroneCI and embracing a mono-repo and Heptapod will deprecate most of it anyway.

Precisely, the tool has created a niche that keeps away potential contributors that despise the tool and would vouch for a change. Current Tryton contributors and the use of Mercurial are a circular phallacy.

That said, I think Heptapod is a huge step in the right direction. Lately I have been interacting with some projects in that plaform and I look forward to seeing Tryton hosted there. Is there any update in this regard?

15

I think that if people that want to contribute are rebutted by the fact of using a different DVCS, they are not worth the pain of migrating to git.

When I need to submit a patch on a project using github, I use github ; they use gitlab ? OK I’ll use gitlab. SQLite is using their own DVCS and it didn’t prevent them to be a big success (and if I have a fix for SQLite one day, I’ll use fossil).

All in all, the DVCS should be a minor point when choosing to use / contribute to a project.

16

Totally, it should be a minor point. Yet, the platforms have a huge impact on new contributor engagement. Tryton’s current platform is specially cumbersome, and I hope moving to Heptapod will be a boost for the community regardless of the underlying VCS.

2 Likes
17

We decided to not rewrite the history because the disadvantage (of loosing changeset reference) is much bigger than the gain of create links with the issue (which could be found by copying the id).

We decided to use: -

We already have one working but we may wait for the next mercurial release to reduce the number of commit on closed branch because it is not allowed to commit with close option on a closed branch. So we have to do a merge and than another commit.
The conversion last for 7:30.

18

Also the easy contributor workflow is getting worked on, it will be integrated before the end of the year but probably will not be there when we do the switch.

19

Today we pushed the converted repository on https://foss.heptapod.net/ and start working on configuring the CI. We have succeeded to launch tests for trytond and country module using PostgreSQL as service.
The next week we will work on detecting which tests must be run.

Our initial tests on this topic seems to be working.

1 Like
20

For the migration of the bugtracker, we can convert the component into labels.
I think it is less interesting in a mono-repository to use the module name as label. I think we should use the same set of names as for the discourse tags.
So I wrote a script that convert the current components into labels:

#!/bin/env python

import sys

rename = {
    'ldap_authentication': 'authentication_ldap',
    'ldap_connection': 'authentication_ldap',
    'neso': 'tryton',
    'google_maps': 'gis',
    'google_translate': 'trytond'
    }
labels = {
    'account': "accounting",
    'analytic': "accounting",
    'chronos': "timesheet",
    'sale': "sales",
    'purchase': "purchasing",
    }


def convert(name):
    name = rename.get(name, name)
    prefix = name.split('_', 1)[0]
    if name.startswith('web'):
        prefix = '-'.join(name.split('_', 2)[:2])
    elif name.startswith('trytond_'):
        prefix = name.split('_', 2)[1]
    return labels.get(prefix, prefix.lower())


if __name__ == '__main__':
    print(convert(sys.argv[1]))