Import csv data from trytond-admin

Feature Ideas
1

Hello,

wouldn’t it be convenient if, after creating a database with trytond-admin, we had a new option to load data from a csv file in the same tool ?

1 Like
2

Hi, is an interesting Idea.

But, What is the need to do this?, Tryton provided a method for import and export information from the clients in csv.

3

The use is to automate setup of pre-populated databases by the system administrator.
Automation provides several advantages:

  • setup of pre-populated databases by the system administrator.
  • to reproduce the same import process, in the right order, of consistent data.
  • consistent updates of referential data as customs export categories, or maybe suppliers list-prices and so on.
  • and of course, when you need to update a large number of databases.

There already exist several tools to do this in the community. The point is to integrate it to tryton-admin with the model_name and the path to the csv file in order to standardize this process.
Of course, it would activate the built-in CSV import function.

4

I wrote this script in other situation trying to implemented your idea. Maybe could be a useful. Is based in tryton scripts to import countries and currencies.

from argparse import ArgumentParser
import sys
import os
import csv


try:
    import argcomplete
except ImportError:
    argcomplete = None

try:
    from proteus import Model, config
except ImportError:
    prog = os.path.basename(sys.argv[0])
    sys.exit("proteus must be installed to use %s" % prog)

try:
    from tqdm import tqdm
except ImportError:
    tqdm = None


class TrytonImportScript():

    def __init__(
            self,
            database: str,
            config_file: str,
            model: str,
            source_file: str,
            unique_key: str = None):
        if not database or not model or not source_file:
            raise ValueError((
                "Database name, model name(__name__)"
                " and source file are mandatory"
            ))
        self.db = database
        self.model = model
        self.source_file = source_file
        self.config_file = config_file

    def get_model(self):
        """Returns Tryton model to import records"""
        return Model.get(self.model)

    def get_records(self):
        """
        Returns existing records in the trryton model to import records
        """
        return self.get_model().find([])

    def _progress(self, iterable, **kwargs):
        if tqdm:
            return tqdm(iterable, disable=None, **kwargs)
        else:
            return iterable

    def update_records(self, current_records):
        print(f"Update {self.get_model().__name__}", file=sys.stderr)
        BaseModel = self.get_model()
        records = self.open_source_file()
        new_records = []

        for record in self._progress(records):
            try:
                new_record = BaseModel()
                for attr, value in record.items():
                    setattr(new_record, attr, value)
                new_records.append(new_record)
            except ValueError as e:
                raise Exception(e)

            BaseModel.save(new_records)
        return {c.code: c for c in new_records}

    def open_source_file(self):
        try:
            with open(self.source_file, newline='') as csvfile:
                reader = csv.reader(csvfile, delimiter=',')
                header = next(reader)
                return [{
                    l: n for n, l in zip(row, header)
                } for row in reader]
        except ValueError as e:
            raise ValueError(e)

    def main(self):
        config.set_trytond(self.db, config_file=self.config_file)
        with config.get_config().set_context(active_test=False):
            self.do_import()

    def do_import(self):
        current_records = self.get_records()
        records = self.update_records(current_records)

        return records

    def run(self):
        parser = ArgumentParser()
        parser.add_argument(
            '-d', '--database', dest='database', required=True)
        parser.add_argument(
            '-c', '--config', dest='config_file',
            help='the trytond config file')

        if argcomplete:
            argcomplete.autocomplete(parser)

        args = parser.parse_args()
        self.main(args.database, args.config_file)
5

Thank you for the proposal for a generic tool. It may help some of us.
The initial question was not about creating such a script, but about using trytond-admin to process imports.
I’d like to get arguments “against” also.

1 Like
6

If such option is implemented, I think it should manage:

  • import multiple files
  • manage the order the files are imported
  • allow to have different files for the same model
  • be transactional (all or nothing)
2 Likes