Using subrepository

(Cédric Krier) #1

Currently we are using hgnested to ease development of all repositories. As the maintainer of hgnested, I start to be tired of burden of maintaining it over each new Mercurial release because the internal API change almost at each release.
Also hgnested does not ensure the coherence of the nested tree. For example, it does not clone new repositories, it fails when switching to an other branches that a nested repository does not have etc.

So I propose to use the subrepository builtin feature of Mercurial which is more mature now than when we started hgnested and which is standard feature.
The main repository which I propose to name development will contain this subrepo structure:

  • trytond
  • tryton
  • sao
  • proteus
  • modules
    • account
    • party
    • …

The repository will contain a hook that manage symlink inside trytond/modules (must create missing and remove unknown based on .hgsub configuration). This hook could be added to the update operation.

The workflow will be that every development on default branch should be committed from the development repository. This will ensure to update the .hgsubstate (in case of mistake, the developer will have to manually update the development repository with the latest revision of all subrepo).

The back-port to older branches will be pushed on each individual repository and when everything have been back-ported, the branches of development repository will be updated.

The release process will stay as-is but a new branch will be created for the new series on the development repository and the default branch will be updated.

For the review, individual patches may be uploaded to Rietveld (or nested one). When we will have a patch mailing list, individual patch could be sent using patchbomp or a manual diff with the option --subrepos from development repository.

Optionally, we could use this new repository to trigger a build that runs all the tests.

Newsletter August 2018
(Vincent Bastos) #2

It makes sense to use another existing tool. It does seem a little more effort to use than hgnested. It might be worth running a trial on the side to run all the functions that are currently being used with hgnested.

(Sergi Almacellas Abellana) #3

I’m not sure if this is really required. If the modules is directory is a symlink to trytond/trytond/modules and we checkout the modules into the module directory it will work without problems.

(Cédric Krier) #4

I do not understand what it means.

subrepository can not be in a subrepository except if it is defined as a subrepository in the subrepository. But we do not want that trytond contains all the modules as subrepositories because we do not want to have to commit in trytond because a module changed.

(Sergi Almacellas Abellana) #5

Then we need the hook as you described.

(Cédric Krier) #7

I understand that mercurial API’s need to evolve but this create too much work too follow the changes for a small extension like hgnested. Especially if standard alternative exists now that provides even more features.

I know but I think it applies to our use case. hgnested is probably also a last resort extension.

As far as I can see it was about using Git sub-repository. Indeed if Mercurial had no security issues, it will be suspicious. But having some, means that people are auditing it and so there are less issues. :wink:
Indeed we do not know if hgnested does not have security issues.

Have you another proposal?

(Cédric Krier) #8

I started the implementation at Issue 7577: Setup subrepository - Tryton issue tracker
I named it tryton-env like that at least on a default clone, the folder will contain the name tryton.

(Cédric Krier) #9

The repository is published at tryton-env: log
The .hgsub is ordered for the modules by alphabetic order, it is better for maintainability to keep the order.
I propose to all developers to push using it for few weeks to see if the workflow works correctly. If it is the case, we will deprecate hgnested on the website and for advise tryton-env.

(Cédric Krier) #10

I had to recreate the tryton-env repository because I used ../ for the target in .hgsub to reach the canonical repositories. But unfortunately, the prevent to clone locally the repository (if you do not reproduce the same structure). I found this was very annoying for work, so I recreated it using local path and configured hgweb and mercurial-server to serve the repositories from inside tryton-env repository.
If you have already cloned the repository, you must delete it and re-clone a new one. Sorry for the inconvenient.

(Cédric Krier) #11

The command hg import does not work to import patch on tryton-env for subrepository by default. But it works if the ui.patch=patch.

I’m working on a patch that will make upload.py to activate subrepo option by default.

(Cédric Krier) #12

Here is the PR: https://github.com/rietveld-codereview/rietveld/pull/568

(Vincent Bastos) #13

I noticed that in some recent patched uploaded that they seem to be from the master repo.

e.g. 51361002 - https://codereview.tryton.org/51361002/#ps160001

Shouldn’t patches for modules be of the subrepo not the master?

(Cédric Krier) #14

It adds a new module that should be registered in .hgsub so I find it simpler to do like that.

(Cédric Krier) #15

I personally made all the operations I used to do: add single feature, make global changes, back-port patches and make releases. I must say that the subrepo workflow is pretty convenient for me. So for me, the experiment is a success.
To fully migrate, we still need:

Optional:

(Nicolas Évrard) #16

It won’t work because review.py use shell commands to get the information from mercurial whereas hgreview directly plugs into mercurial internals. Unfortunately the function hgreview is using to get the modified / added / deleted files works can be executed on subrepos but do not return this data (https://www.mercurial-scm.org/repo/hg/file/tip/mercurial/context.py#l359).

Of course using itersubrepos is an option I plan to implement but not right now.

(Cédric Krier) #17

Done: http://git.tryton.org/?p=codereview.tryton.org.git;a=commit;h=31c400bb0e69ed3602a4e636ba9e81876ff13266

(Cédric Krier) #18

I’m wondering what is the real benefit of being an extension?
For me, the current advantages are:

  • Use an editor for the message
  • Store the new issue id
  • Use the server from the mercurial configuration

I think those features could be implemented in upload.py and publish on review without too much work.

And for those who want to remove issue id on commit, they can register a hook like this:

[hooks]
commit = rm -f .hg/review_id
(Sergi Almacellas Abellana) #19

I’m using upload.py and those two features are the ones which I’m missing.

I have a local modified version of upload.py which changed the default server to upload, so not really missing the configuration too much as most of the time I only work with one codereview server.

Agree that implementing on upload.py will be better. I’m currently using upload.py because hgreview does not work on latest py3

(Nicolas Évrard) #20

That’s why I say that I’ll implement it but probably not right now.
In the end it might be easier to just use upload.py