New structure for the tryton modules

Feature Ideas
1

At B2CK when developing an implementation for our customers we usually create one module with a bunch of backports of some issues, some customisations, etc.

It’s been some years that we’ve practiced this way and it works kind of great but it could be better.

  • There is this requirement to repeat the name of module when invoking Pool.register
  • The XML files must be specified in the tryton.cfg

We’ve discussed with @ced how we could make it better and we’ve got some ideas that require some development (not much) but we’d like to receive some input from the community.

The idea is that in the tryton.cfg we would have a new section (submodules maybe) that would contain a list of sub-directory names that contain the submodule code and XML.

What would be cool would be that when specifying name of a view or the path of an icon, it’s not required to use the name of the submodule.

Which means that this XML:

        <record model="ir.ui.view" id="organization_view_list">
            <field name="model">country.organization</field>
            <field name="type">tree</field>
            <field name="name">organization_list</field>
        </record>

in a submodule named sub would reference the file module/view/sub/organization_list.xml. But I’m not sure that’s doable so we might have to settle to use XML entities or even Genshi not to repeat ourselves too much.

For the translations unfortunately as all the translations of a module are currently exported as a unique file they are not suited to be put into the submodule directory.

An idea that was rejected was to have view and icons directory into the submodule. The rational for this is that it would be difficult to explain to newcomers the difference between a module and a submodule if their structure is too much alike. With the same reasoning even if I named the configuration file of the submodule tryton.cfg, I’m not sure that it should be named like that.

Here’s the structure we could have:

tryton-module
├── __init__.py
├── code.py
├── code.xml
├── tryton.cfg
├── icons
│   ├── cool_icon.svg
│   └── sub_module
│       └── even_cooler_icon.svg
├── locale
│   ├── de.po
│   └── fr.po
├── sub_module
│   ├── __init__.py
│   ├── extension.py
│   ├── extension.xml
│   ├── tryton.cfg
│   └── tests
│       ├── __init__.py
│       ├── scenario_extension.rst
│       ├── test_module.py
│       └── test_scenario.py
├── tests
│   ├── __init__.py
│   ├── scenario.rst
│   ├── test_module.py
│   └── test_scenario.py
└── view
    ├── form.xml
    ├── list.xml
    └── sub_module
        ├── form_extension.xml
        └── list_extension.xml

What do you think about that? Would you adopt this structure for your own customization?

2

I also create one big module, but after getting lost, move parts in their own module. So submodules seems a good idea but I have my doubts about the proposed solution. It reminds me at the early EmberJS times where all the different parts also were scattered over the different directories. Users didn’t like that and the structure was changed so that all the parts were in the same directory they belong.

That’s kind of the same EmberJS talk … “Why HTML templates in different places? Let’s put them in the same place in subdirectories”, which confused a lot of users because you have to click and search around a lot and for the bigger apps it became a nightmare. So please make the submodule where all the parts needed by the submodule are in that directory.
Explanation why you want to use submodules is IMO very clear, to distinguish and make things more clear. For example you want to customize the party module. You add a submodule called party and add your Python code and XML, but also the subdirectory view because you want to customize the form of the party. Also newcomers can be trained.

Harsh answer: No, not in the proposed form. It adds to much overhead because of the scattered files.

My biggest issue is that you can’t specify in Tryton where your modules live. At the moment it is always in the modules directory, but it would be awesome to have the ability to add other modules directories. This will also ease the possibility to add new modules to your containers by adding a volume.

3

I do not think we need to change the tryton.cfg.
Indeed the tryton.cfg could just contain an entry to load the main XML file in the subdirectory. This main XML should use the include XML syntax to define the other XML files to load.

For me it will be great to use XML template from Genshi to have:

<tryton py:with="submodule='sub_module'">
    <data>
        <record model="ir.ui.view" id="organization_view_list">
            <field name="model">country.organization</field>
            <field name="type">tree</field>
            <field name="name">${submodule}/organization_list</field>
        </record>
    </data>
</tryton>

I hope most of developers do not click and search :slight_smile:

This is too much change from the view loading mechanism. Also it will require to store in the view record the “submodule”. But I’m strongly against leaking the “submodule” design in the core of trytond.

It will never happen because we need to enforce uniqueness of module.

This would be a very bad idea as you can not manage the dependencies correctly.

4

I’m missing a solution for that.
For me we should remove the module name in the Pool.register call. For that we should have a context manager that define which module is currently registering. So we could simply nest the call to Pool.register without having to carry the module name everywhere.
Of course calling Pool.register outside the context manager should fail.

5

Not need for Genshi to include XML xmlns:xi="http://www.w3.org/2001/XInclude" should work.

6

Yeah it’s exactly what I was afraid of.

Well with entrypoints you can have the module where you want.

Failing to do that would miss the point I think.

XML files in subdirectory can already be loaded. And it’s easier to do it in the tryton.cfg than using xi:include (but if it works, it’s nice to have it).

We don’t need Genshi for that as XML entities work already.
Moreover, using the Genshi machinery for each file (or having something to discriminate between the files requiring it or not) is complicating the code for little gain.

I have a bad news for you :smiley:.

Because you’re seeing that as the “submodule design” instead of seeing it as the path of the directory containing the view file.

I don’t understand. Using entrypoints people can already put the code where they want (but they have to install the python package which as a nice side effect should take care of the dependencies at the same time).

After more testing it’s the only thing left. Sure we could fix that but for now we can live with that.

7

I do not agree. For me the burden when managing “subfolders” is that you have to register all the XML files of the subfolder to tryton.cfg. But if you can just register one XML which will load all the others, this solves the problem without changing the syntax of tryton.cfg.

I do not understand.

I do not understand. What are you calling “XML entities”?

For me there is a huge gain in supporting a templating language for the data XML.

It enforce uniqueness.

I do not see the point to add partial support if “subfolder” cannot be moved or renamed easily.

8

Yes, for us this would be a great feature, too.

Usually we start a custom project with just one custom module. But as the feature requests getting more and more, we get lost which code depends to what feature. We also think of the general re-usability of a feature and in which module to put it. Our questions are always:

  • Is there a module which can be enhanced? → existing_module
  • Is the functionality worth a new module? → new_module
  • Is the functionality just custom? → custom_module

But the decision we make is a kind of guessing, because we can not foresee our future project opportunities and which functionality will be re-used by others.

For us submodules would be comfortable. Because we can always start with a custom module AAA for a project. And once another project custom module BBB requires the functionality from AAA, we can easily copy it. If then a custom project CCC also requires the functionality from AAA, we can think of moving the functionality into its own module.

9

In the past the XML and the Python code of a Tryton module was collected in the module directory. Me and other programmers hated it, because everything XML related was mixed together in one file: view definitions, access rules, menu entries, data… For me it is good to have the views separated and collected together in a folder.

But AFAIK, today it is no problem to use the ‘old’ style without the views/ directory and collecting all the views and other XML definitions of a model in one file.

10

Thanks to a sponsoring of Coopengo I’ve been working on this feature.

We’re going to introduce a new option in the tryton section of the config file: include. The value of this option is a list of names, each name represents a subdirectory containing its own tryton.cfg.

A new path attribute in XML files will allow to specify that the value of the field is a path relative to the directory of the current tryton.cfg being processed.

As of now, the design still puts the view files in the view directory at the root of the module but under the same hierarchy as the tryton.cfg.

Technicaly it’s not difficult to do differently as it’s just a matter of splitting the value of the path between its basename and dirname component and using those to rebuild the real path of the view file. But doing so would be an issue when packaging the module.

I will also introduce a separate merge request that intend to rationalize the way the reports and icons are found by the code. In the future we will make it mandatory to put those in icons and reports directories. It might help having less clutter in the module.

And finally we’d like to make mandatory to use icons and reports directories as the place to store the reports and icons.

1 Like
11

For me it should prepend the relative path from the module to the XML file to the value of the tag.
It should not be relative to the place where the tryton.cfg loading this XML is placed.

12

It’s in fact what the code does.

13

If I make a sub-folder, I usually prefer to have everything in there. So views / icons / reports folders in my sub-folder.

account_invoice
  +- my-sub-module
  |    +- icons
  |    +- views
  |    |    + my_view.xml
  |    +- __init__.py
  |    +- my_sub_module.py
  |    +- my_sub_module.xml
  |    +- tryton.cfg
  +- views
  +- __init__.py
  +- my_module.py
  +- my_module.xml
  +- tryton.cfg

Including test scenarios in sub-folder is also possible, so it makes a folder a rather independent part of the main module, which is one of my use cases.

1 Like
14

But this is the main incomprehension, there is no “sub-module” but only file split for organisation. The logical unit is still the module.

15

Yes it is.

I guess that it depends on the use case for those folders. I actually have three at the top of my head:

  • Same module, but a big feature that I may want to put in a separate module later on
  • Managing an extras_depend with a lot of adapatations
  • Merging the contents of another module

I think that in those three cases at least, having everything related in a separate folder makes sense, because when working on it you want to think about it as a separate feature. The fact that this feature is part of a bigger module is more a packaging problem.

If you see it as a way to organize files (like in account, where we have those very big files that could be split a little, but that would make 50 py files in the root folder), your approach makes more sense indeed.

My point is that I think the target structure heavily on the use case. I know for sure that I have the first three above, but the second approach is more restricted to very big modules imho. Also, I think it is mostly useful for big modules, and I think it can already (mostly) be covered without the include feature we are talking about.

16

If they are independent and you want them to be independent, you must create independent modules.

I do not understand what is the packaging problem.

17

I agree with Jean here. I find having views/icons/reports in sub folder more intuitive.

Our use case is that we usally backport modules that are developed in upstream to older version in custom modules. In such case, we prefer to have everything in a single folder because when we upgrade to newer series we just need to delete the entire directory.

Of course its not a big deal as running test will complain about unused views, but when modifing the module its easier to work on the subdirectory and open everything from there (as you will do in a smaller tryton module) than need to navigate to another folder to edit view files.

18

If I was not clear about this earlier, I wholly agree here.

“sub-module” is not the right name here, it introduces a confusion.

I see it as a “feature” / “extras_depend” thing. I toyed with the idea of enforcing a dedicated folder per “extras_depend” combination, so that people not familiar with the module are not confused with multiple class declarations for the same model in the same file.

I think “extras_depend” makes a good case here. We could indeed create a separate module to create the “link” between to modules (like we used to do a few years back), and it would have a full module structure (with a views folder, a tests folder, etc…)

We agree, I think, that “extras_depend” related content in a module is not a “sub-module”. However, mixing it everywhere in the main folder is not ideal from a readability point of view. Having it all neatly packed in a separate folder just make it easier to work with.

Like I said earlier, your use case is valid, and the file structure you suggest definitely makes sense for it. I just think that:

  • It is more or less already solved with what already exists, and as far as I understand the proposed change will not make it impossible to split python / xml files in folders
  • It does not fit a real use case we have multiple instances of in our codebase (and other’s it seems)
19

Backporting modules is better done as a module otherwise you will have issue like duplicate button registration etc.

Well that can be done with rm -rf */sub-directory/. Not really complicated.

Good catch the test_view will be much more complex if we allow to put views anywhere.

Navigation?
I do not find more complicated nor longer to type:
:e modules/my-module/view/my-submodule/file.xml
than:
:e modules/my-module/my-submodule/view/file.xml

Indeed the extras_depend is a nice use case of the feature.
But I think we will go even further, I would like to have predictable path for any object declaration based on their __name__. Thus we may end up having a replicate of the modules structure inside each module.

I do not understand.

20

I may be wrong, but testing views requires that all modules (including extras) be loaded to work properly?
I would rather have the tests of a sub-folder include the views of this folder, and the “global” test (which would run without the extras as they would be handled in sub-folders) those in the root folder.

This is very workflow dependent. People using visual navigation (and there are a lot of them) are much more dependent of the folder structure for edition.

I already thought of it as well, but I am not sure of how it would be really help. What I am afraid of is that it will require a lot of table renaming, so a significant maintenance cost for existing deployments for a limited gain.

I have multiple identified cases in our codebase where having everything in a sub-folder would be better than splitting it. And if I understand correctly, @pokoli as well.