Using a mono-repository


(Cédric Krier) #1

Since a long time, we are looking for the right tools to develop Tryton. Our current solution is commonly accepted to not be the best but more a lack of better. I think the problem is that we search for tools that works better with our current workflow instead of rethinking the all workflow. So I will start by describing my ideal contribution workflow without caring about the tools. The workflow should work for a core developer but also for a casual contributor:

  1. Clone
  2. Modify
  3. Commit
  4. Push
  5. Review (iterate from 2)
  6. Publish (core developer)

How can we achieve such simple workflow.

Mono-repository

The first point is that we should use a single repository which contains the server, the clients and the modules. It should be ready to use/develop. It should be possible to migrate the existing repositories into a single one while keeping the all history and tags. (I have discussed with the guys from Octobus about our case).
There are few points that should be fixed prior the migration:

Tests

We need to find a solution to run tox only for the modules which were changed. But also that tox installed the Tryton’s dependencies from the current clone (no more trydevpi.tryton.org). This could be done with a script in command_pre that parse recursively tryton.cfg for dependencies and install them (but without failing if they can not be found. E.g. running test from installed package).
There is also the .drone.yml which is the same for all repositories except for sao but they could probably be merged with two entries in pipeline.

Tags

We will have to tag each release with a prefix. E.g.: trytond-5.0.1, modules-account-4.8.4 etc.
But I do not think it is a major problem.

Commits

As all changes will go into the same history, it will be needed to prefix the commit by the name of impacted module. E.g.: account: Update tax code lines when updating chart from template.
This may be an inconvenient for module having long name, so we should probably be not too strict or not do it at all (it is possible to get the history only for a directory).

readthedocs

We will keep the build of each module documentation separated. For that the hook that trigger the build should look at the changed files and trigger the right rtd project (using a configuration field module->rtd ID). The rtd project should be configured to point to a conf.py file inside its doc directory (the file could just be empty but it is necessary to define where is the directory in the mono-repository).

Open repository

To ease the push process, it is easier to be allowed to push on the cloned repository. We could setup a repository open to all which keeps changesets as draft. To organize a little bit this open repository, the evolution extension and its topic sidekick can be used to follow development of a feature (aka topic).

Core developers will be in charge to push reviewed topic to the official repository and thus makes the changesets public. Every changes should land first on the open repository before being published to the official repository, this ensures a one direction synchronization.

The open repository will use the ssh keys stored on the bug-tracker profile of the user. This will prevent spam and vandalism.

Testing

The drone instance could be plugged to this open repository and so each changes could be fully tested before being published.

Cleaning

Some cleaning task will be needed to remove topic that were refused.

Bug tracker

The bug tracker hook could also be configured on this open repository. It should just post comment on the linked issue but not close it (this could be based on the phase of the changeset: public=close, draft=testing).

Code Review

As changesets will be committed and pushed to an open repository, it will not more be practical to use Rietveld for code review because it works better as pre-commit review.

Indeed for code review, we just need to be able to comment on an existing changeset. We already have a tool to display the changeset, it is hgweb. We just need to add a feature to comment on this hgweb pages. This is not a new idea as this post shows. After some research, I found that there is a standardization effort to provide web annotation. But this tool may be too bloated for our usage, we could have a simple Javascript file that we embed in the hgweb page and a service to store and retrieve comments. It can also rely on the bug-tracker to notify the nosy list.


(Sergio Morillo) #2

My two cents,
I worked in a project some years ago in which libraries source code (not python project) were managed with a single repository. IMHO the current structure of repository per module is better than a single repository for many reasons (that you have pointed out):

  • isolated commit history and traceability.
  • isolated module patching.
  • managing module releasing.

(Cédric Krier) #3

You can have the history filtered per folder with mercurial: https://www.mercurial-scm.org/repo/hg/help/log

The import command can be applied on subdirectory: https://www.mercurial-scm.org/repo/hg/help/import
Also now with the subrepo, we already work with global patch.

Indeed it does not really affect the developer but only the release manager. And as long as it is automated by script, it is not a real problem.
The only point is that it will not be possible to get out of the repository a set of specific versions (e.g. account==5.0.1, party==5.0.3, etc.) but virtualenv and pip are your friend for that.


(resteve) #4

I understand that idea is same OpenERP or Odoo: an unique repository server and addons.

I like separated repos because we don’t like download all modules and is easy control commits and backports. For example, I like to skip account_es, account_fr, account_fr_chorus, account_payment_stripe… modules. Also list all commits is a bit chaotic [1]

How to skip some modules that don’t like to download in a mono-repository?

Current project workflow, the worst part of workflow is the codereview (configuration and Google privacy). The rest, a good documentation in www.tryton.org is correct (discussion - report bugs - review - translations)

[1] https://github.com/odoo/odoo/commits/12.0


(Sergi Almacellas Abellana) #5

Why you will like to skip some modules?

When developing on tryton if something is changed all the modules should be adapted to the change. Having all the modules in the same repository is a good way to enforce it.

P.S: I’m a little bit worried about why a spanish user skips the Spanish Chart of Accounts :open_mouth: It will be great to hear you rational about it but maybe as part of another topic as it’s unrelated to this one


(Cédric Krier) #6

I think if you are skipping some modules, it means that you are not really developing Tryton but you are developing on Tryton. So you should use package released which gives the control you want.

Then do not look at all the commits. It is possible for example to look at the changes of a file inside a branch: trytond: bin/trytond history

For example Python has way more modules than Tryton and it is a mono-repository. They do not even prefix the commit message and it is quite fine.


(resteve) #7

I remember in the old web said: Tryton is a framework.

You could run trytond with modules available in http://hg.tryton.org/modules/ , or custom modules, or third modules. Modules in hg.tryton.org is not required.

Other modules we not need to download and/or test are: sale_payment, stock_package_shipping_*… != Spanish Chart of Accounts.

Yes and not. In some projects only need to test trytond. In other projects, we need to test trytond + party module (not more modules). And other projects, we add more modules available in http://hg.tryton.org/modules/ . How to download a few modules and not all mono-repository with all modules?

(I ask in dev branch (default) - not available a package release-; not in v5.0, v4.8, … releases).


(Cédric Krier) #8

Yes you can. But if you develop for the Tryton project, you can not skip modules.

Well we do not care about your specific workflow or development. We talk about what suite best for Tryton project development.

As I already said. You are talking about your personal development workflow which we do not care about.
If you want to install just a set of modules than install only what you want. FYI, cloning a mercurial repository is not an installation. Only contributors to Tryton project need to use the repository.

Development branch is only for Tryton development and we care about all our modules.


(Sebastien Marie) #9

@resteve I am unsure to understand the problem you have with mono-repo. You are still free to clone a full repo and install local only a partial sets of modules (cd module && python3 setup.py build -b my/path).

Additionally, it seems it exists a way to do narrow clone (but not necessary officially supported):


(Sebastien Marie) #10

Regarding mono-repository, BSD system have historical worked this way: base system usually carry lot of differents programs, and third-parties applications are maintained in a mono repository too.

Speaking about OpenBSD, ports repository contains 10504 differents applications (the OpenBSD Ports Collection is the infrastructure used to create binary packages for third party applications).


(Cédric Krier) #11

FYI, I have started an experimentation about this workflow with the relatorio project.
I have setup a public repository public/relatorio: log (only accessible for now to committers) which has evolve and topic extension activated.
The repository is linked to drone: https://drone.tryton.org/tryton/public.relatorio
And also to the bug tracker as you can see: https://relatorio.tryton.org/bug49 The hook put the status in pending.
In the topic ‘memory’, there are 3 changesets and I amended two later to add the bug id. The only minor problem I found until now is that the link to the changeset in the older build are no more found by hgweb because they are hidden. It seems that hgweb and phase/topic are not yet fully integrated (e.g. topics are not shown).
I must say that it was nice to be able to break the modification into 3 smaller changes.


(Cédric Krier) #12

I’m wondering if we could not just live with posting comment on the bugtracker with the link to the hgweb page anchor above. It could like like:

public/relatorio: fc0889787c99
Could we use the flush=True?


(Cédric Krier) #13

I found that we can show all changesets with this configuration:

[web]
view = all

So I think it makes sense to activate it on public repositories.


(Sergi Almacellas Abellana) #14

I think this is a good idea! As the link will be clicable will be exactly the same as we have now with emails.


(Cédric Krier) #15

I even think we could have a roundup extension that format link to mercurial with the content of few lines above and below. Of course, it should be cached.


(Sergi Almacellas Abellana) #16

When cloning the public repoisotory the following trace (see the remote error):

hg clone ssh://hg@hg.tryton.org/public/relatorio            
destination directory: relatorio
requesting all changes
adding changesets
remote: error: outgoing.aaaaa_servelog hook raised an exception: filtered revision '277' (not in 'served' subset)
remote: (run with --traceback for stack trace)
adding manifests
adding file changes
added 279 changesets with 597 changes to 74 files
14 new obsolescence markers
new changesets 8ab308134662:c41f8421db10 (3 drafts)
updating to branch default
65 files updated, 0 files merged, 0 files removed, 0 files unresolved

Am I doing something wrong? Or this is the expected behaviour?


(Cédric Krier) #17

I think it is a bug of mercurial-server logging system which fails for revision hidden.


(Cédric Krier) #18

There is an extra difficulty with this because drone uses two different images for cloning and running tests. This means that the image actually running the tests does not have mercurial installed so guessing the changed files may be complicated.
One option could be to have the cloning image storing those information in the workspace but this would require a custom docker image.
Another option could be to have an initial step in the pipeline which uses a docker image with mercurial and generate a file with the changed files. This file will be used by the next step to know which test to run.