Using a mono-repository

Organisation
1

Since a long time, we are looking for the right tools to develop Tryton. Our current solution is commonly accepted to not be the best but more a lack of better. I think the problem is that we search for tools that works better with our current workflow instead of rethinking the all workflow. So I will start by describing my ideal contribution workflow without caring about the tools. The workflow should work for a core developer but also for a casual contributor:

  1. Clone
  2. Modify
  3. Commit
  4. Push
  5. Review (iterate from 2)
  6. Publish (core developer)

How can we achieve such simple workflow.

Mono-repository

The first point is that we should use a single repository which contains the server, the clients and the modules. It should be ready to use/develop. It should be possible to migrate the existing repositories into a single one while keeping the all history and tags. (I have discussed with the guys from Octobus about our case).
There are few points that should be fixed prior the migration:

Tests

We need to find a solution to run tox only for the modules which were changed. But also that tox installed the Tryton’s dependencies from the current clone (no more trydevpi.tryton.org). This could be done with a script in command_pre that parse recursively tryton.cfg for dependencies and install them (but without failing if they can not be found. E.g. running test from installed package).
There is also the .drone.yml which is the same for all repositories except for sao but they could probably be merged with two entries in pipeline.

Tags

We will have to tag each release with a prefix. E.g.: trytond-5.0.1, modules-account-4.8.4 etc.
But I do not think it is a major problem.

Commits

As all changes will go into the same history, it will be needed to prefix the commit by the name of impacted module. E.g.: account: Update tax code lines when updating chart from template.
This may be an inconvenient for module having long name, so we should probably be not too strict or not do it at all (it is possible to get the history only for a directory).

readthedocs

We will keep the build of each module documentation separated. For that the hook that trigger the build should look at the changed files and trigger the right rtd project (using a configuration field module->rtd ID). The rtd project should be configured to point to a conf.py file inside its doc directory (the file could just be empty but it is necessary to define where is the directory in the mono-repository).

Open repository

To ease the push process, it is easier to be allowed to push on the cloned repository. We could setup a repository open to all which keeps changesets as draft. To organize a little bit this open repository, the evolution extension and its topic sidekick can be used to follow development of a feature (aka topic).

Core developers will be in charge to push reviewed topic to the official repository and thus makes the changesets public. Every changes should land first on the open repository before being published to the official repository, this ensures a one direction synchronization.

The open repository will use the ssh keys stored on the bug-tracker profile of the user. This will prevent spam and vandalism.

Testing

The drone instance could be plugged to this open repository and so each changes could be fully tested before being published.

Cleaning

Some cleaning task will be needed to remove topic that were refused.

Bug tracker

The bug tracker hook could also be configured on this open repository. It should just post comment on the linked issue but not close it (this could be based on the phase of the changeset: public=close, draft=testing).

Code Review

As changesets will be committed and pushed to an open repository, it will not more be practical to use Rietveld for code review because it works better as pre-commit review.

Indeed for code review, we just need to be able to comment on an existing changeset. We already have a tool to display the changeset, it is hgweb. We just need to add a feature to comment on this hgweb pages. This is not a new idea as this post shows. After some research, I found that there is a standardization effort to provide web annotation. But this tool may be too bloated for our usage, we could have a simple Javascript file that we embed in the hgweb page and a service to store and retrieve comments. It can also rely on the bug-tracker to notify the nosy list.

2

My two cents,
I worked in a project some years ago in which libraries source code (not python project) were managed with a single repository. IMHO the current structure of repository per module is better than a single repository for many reasons (that you have pointed out):

  • isolated commit history and traceability.
  • isolated module patching.
  • managing module releasing.
3

You can have the history filtered per folder with mercurial: Help: log

The import command can be applied on subdirectory: Help: import
Also now with the subrepo, we already work with global patch.

Indeed it does not really affect the developer but only the release manager. And as long as it is automated by script, it is not a real problem.
The only point is that it will not be possible to get out of the repository a set of specific versions (e.g. account==5.0.1, party==5.0.3, etc.) but virtualenv and pip are your friend for that.

4

I understand that idea is same OpenERP or Odoo: an unique repository server and addons.

I like separated repos because we don’t like download all modules and is easy control commits and backports. For example, I like to skip account_es, account_fr, account_fr_chorus, account_payment_stripe… modules. Also list all commits is a bit chaotic [1]

How to skip some modules that don’t like to download in a mono-repository?

Current project workflow, the worst part of workflow is the codereview (configuration and Google privacy). The rest, a good documentation in www.tryton.org is correct (discussion - report bugs - review - translations)

[1] https://github.com/odoo/odoo/commits/12.0

5

Why you will like to skip some modules?

When developing on tryton if something is changed all the modules should be adapted to the change. Having all the modules in the same repository is a good way to enforce it.

P.S: I’m a little bit worried about why a spanish user skips the Spanish Chart of Accounts :open_mouth: It will be great to hear you rational about it but maybe as part of another topic as it’s unrelated to this one

6

I think if you are skipping some modules, it means that you are not really developing Tryton but you are developing on Tryton. So you should use package released which gives the control you want.

Then do not look at all the commits. It is possible for example to look at the changes of a file inside a branch: trytond: bin/trytond history

For example Python has way more modules than Tryton and it is a mono-repository. They do not even prefix the commit message and it is quite fine.

7

I remember in the old web said: Tryton is a framework.

You could run trytond with modules available in http://hg.tryton.org/modules/ , or custom modules, or third modules. Modules in hg.tryton.org is not required.

Other modules we not need to download and/or test are: sale_payment, stock_package_shipping_*… != Spanish Chart of Accounts.

Yes and not. In some projects only need to test trytond. In other projects, we need to test trytond + party module (not more modules). And other projects, we add more modules available in http://hg.tryton.org/modules/ . How to download a few modules and not all mono-repository with all modules?

(I ask in dev branch (default) - not available a package release-; not in v5.0, v4.8, … releases).

8

Yes you can. But if you develop for the Tryton project, you can not skip modules.

Well we do not care about your specific workflow or development. We talk about what suite best for Tryton project development.

As I already said. You are talking about your personal development workflow which we do not care about.
If you want to install just a set of modules than install only what you want. FYI, cloning a mercurial repository is not an installation. Only contributors to Tryton project need to use the repository.

Development branch is only for Tryton development and we care about all our modules.

9

@resteve I am unsure to understand the problem you have with mono-repo. You are still free to clone a full repo and install local only a partial sets of modules (cd module && python3 setup.py build -b my/path).

Additionally, it seems it exists a way to do narrow clone (but not necessary officially supported):

10

Regarding mono-repository, BSD system have historical worked this way: base system usually carry lot of differents programs, and third-parties applications are maintained in a mono repository too.

Speaking about OpenBSD, ports repository contains 10504 differents applications (the OpenBSD Ports Collection is the infrastructure used to create binary packages for third party applications).

11

FYI, I have started an experimentation about this workflow with the relatorio project.
I have setup a public repository http://hg.tryton.org/public/relatorio/ (only accessible for now to committers) which has evolve and topic extension activated.
The repository is linked to drone: https://drone.tryton.org/tryton/public.relatorio
And also to the bug tracker as you can see: https://relatorio.tryton.org/bug49 The hook put the status in pending.
In the topic ‘memory’, there are 3 changesets and I amended two later to add the bug id. The only minor problem I found until now is that the link to the changeset in the older build are no more found by hgweb because they are hidden. It seems that hgweb and phase/topic are not yet fully integrated (e.g. topics are not shown).
I must say that it was nice to be able to break the modification into 3 smaller changes.

12

I’m wondering if we could not just live with posting comment on the bugtracker with the link to the hgweb page anchor above. It could like like:

public/relatorio: fc0889787c99
Could we use the flush=True?

13

I found that we can show all changesets with this configuration:

[web]
view = all

So I think it makes sense to activate it on public repositories.

14

I think this is a good idea! As the link will be clicable will be exactly the same as we have now with emails.

15

I even think we could have a roundup extension that format link to mercurial with the content of few lines above and below. Of course, it should be cached.

16

When cloning the public repoisotory the following trace (see the remote error):

hg clone ssh://hg@hg.tryton.org/public/relatorio            
destination directory: relatorio
requesting all changes
adding changesets
remote: error: outgoing.aaaaa_servelog hook raised an exception: filtered revision '277' (not in 'served' subset)
remote: (run with --traceback for stack trace)
adding manifests
adding file changes
added 279 changesets with 597 changes to 74 files
14 new obsolescence markers
new changesets 8ab308134662:c41f8421db10 (3 drafts)
updating to branch default
65 files updated, 0 files merged, 0 files removed, 0 files unresolved

Am I doing something wrong? Or this is the expected behaviour?

17

I think it is a bug of mercurial-server logging system which fails for revision hidden.

18

There is an extra difficulty with this because drone uses two different images for cloning and running tests. This means that the image actually running the tests does not have mercurial installed so guessing the changed files may be complicated.
One option could be to have the cloning image storing those information in the workspace but this would require a custom docker image.
Another option could be to have an initial step in the pipeline which uses a docker image with mercurial and generate a file with the changed files. This file will be used by the next step to know which test to run.

19

I think with a mono-repository, we could probably think about using hosting tool like Kallithea which provides simply pull request workflow with code review tool. But the workflow should be analyzed and tested to ensure it will work for us.

20

I had a new idea for codereview. It is inspired by the discourse code review plugin which is unfortunately linked on to Github. We could use discourse. Indeed discourse is designed for discussion and it can display nicely diff, see below.

example of patch 
# HG changeset patch
# User Cédric Krier <ced@b2ck.com>
# Date 1580837530 -3600
# Node ID 0cbfedbcf98e19b52fed87e495ab00b3b446520c
# Parent  0ba6dd7d1c81c0d29464f522bc92f93494fbd078
Add tests for types checking in PYSON Greater

issue9002
review276711002

diff -r 0ba6dd7d1c81 -r 0cbfedbcf98e trytond/tests/test_pyson.py
--- a/trytond/tests/test_pyson.py	Wed Jan 29 00:05:32 2020 +0100
+++ b/trytond/tests/test_pyson.py	Tue Feb 04 18:32:10 2020 +0100
@@ -224,6 +224,8 @@
         if not sys.flags.optimize:
             self.assertRaises(AssertionError, pyson.Greater, 'test', 0)
             self.assertRaises(AssertionError, pyson.Greater, 1, 'test')
+            self.assertRaises(
+                AssertionError, pyson.Greater, pyson.Eval('foo'), 0)
 
         self.assertEqual(pyson.Greater(1, 0).types(), set([bool]))
 
@@ -253,6 +255,10 @@
 
         self.assertEqual(repr(pyson.Greater(1, 0)), 'Greater(1, 0, False)')
 
+        eval = pyson.PYSONEncoder().encode(
+            pyson.Greater(pyson.Eval('i', 0), 0))
+        self.assertTrue(pyson.PYSONDecoder({'i': 1}).decode(eval))
+
     def test_Less(self):
         'Test pyson.Less'
         self.assertEqual(pyson.Less(0, 1).pyson(), {

trytond: 0cbfedbcf98e

Discourse has an API that can be used to post the changeset received to a dedicated category. We could write a hook that use this API to post on a topic per mercurial topic name. We could request that the topic name uses an issue number to ensure uniqueness. When the topic evolves (e.g. rebased or amended), the hook report on the same topic the new set of patches. The reviewbot could also be adapted to post on discourse.
Another option would to customize (or push upstream) a hgweb template that support oEmbed or Open Graph for onebox.