)
7140 0.026 0.000 0.040 0.000 /home/nicoe/projets/tryton/tryton-env/FieldGetC/trytond/trytond/model/modelstorage.py:1493(not_cached)
841/630 0.002 0.000 0.030 0.000 {method 'join' of 'str' objects}
3360 0.006 0.000 0.017 0.000 /home/nicoe/.local/share/virtualenvs/tryton-trunk/lib/python3.8/site-packages/sql/ __init__.py:479(_format_column)
210 0.006 0.000 0.015 0.000 /home/nicoe/.local/share/virtualenvs/tryton-trunk/lib/python3.8/site-packages/sql/ __init__.py:591(params)
210000 0.014 0.000 0.014 0.000 {method 'add' of 'set' objects}
14490 0.007 0.000 0.011 0.000 /home/nicoe/projets/tryton/tryton-env/FieldGetC/trytond/trytond/model/model.py:377(_contains)
3782 0.003 0.000 0.008 0.000 /home/nicoe/.local/share/virtualenvs/tryton-trunk/lib/python3.8/site-packages/sql/ __init__.py:1306( __str__ )
3360 0.003 0.000 0.008 0.000 /home/nicoe/projets/tryton/tryton-env/FieldGetC/trytond/trytond/model/fields/field.py:362(sql_type)
7140 0.006 0.000 0.008 0.000 /home/nicoe/projets/tryton/tryton-env/FieldGetC/trytond/trytond/model/modelstorage.py:1500(to_load)
210 0.007 0.000 0.007 0.000 {built-in method fromkeys}
420 0.005 0.000 0.007 0.000 /home/nicoe/.local/share/virtualenvs/tryton-trunk/lib/python3.8/site-packages/sql/ __init__.py:496(_window_functions)
3992 0.002 0.000 0.007 0.000 /home/nicoe/.local/share/virtualenvs/tryton-trunk/lib/python3.8/site-packages/sql/ __init__.py:249(alias)
210 0.001 0.000 0.006 0.000 /home/nicoe/.local/share/virtualenvs/tryton-trunk/lib/python3.8/site-packages/sql/ __init__.py:262(select)
1.2678403660247568
```
Due to limitations in the profiler, the call to ` __get__ ` is not present (but if I add a call that will just make a call to `super()` the results would be quite like the results of the previous point).
The `timeit` figures is a bit slower than the pure python implementation, I attribute that to some overhead that is probably added by the way I implemented the `Field` and its accelerator class.
# Conclusion
It’s quite hard to find some place where writing some part of the implementation in C would make a big difference.
In fact retrospectively I think that good candidates are part of the framework that could be “self-contained”: by not being too tied to the other part of Trytond, they could be completely implemented in C thus minimizing the need to use the Python datastructures (to some extent of course).
A good candidate would be our cache implementations as they could be quite independent of the rest and yet the do appear at the top of the methods that are slow (not THAT slow but a bit slow anyway).
---
### Author:  [@JCavallo](https://discuss.tryton.org/u/JCavallo)
#### Post date: [January 21, 2021, 4:01pm UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/2 "2021-01-21T16:01:39Z")
Hi,
From working on “performances” related problem rather frequently, I think there may be a misinterpretation, probably caused by CPU throttling / other processes.
If you compare other calls, they all took longer anyway, so it might be worth repeating the test multiple times, ideally by forcing the CPU stepping so it does not change.
(for instance, the second line was 4-5 times in the second call)
Maybe you already did that, but I cannot see how rewriting in C would slow the call (unless the implementation is flawed, which I honestly did not check)
---
### Author:  [@ced](https://discuss.tryton.org/u/ced)
#### Post date: [January 21, 2021, 4:11pm UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/3 "2021-01-21T16:11:49Z")
Indeed instead of using `timeit`, `pyperf` should be used.
[https://twitter.com/VictorStinner/status/1349666308682612736](https://twitter.com/VictorStinner/status/1349666308682612736)
---
### Author:  [@nicoe](https://discuss.tryton.org/u/nicoe)
#### Post date: [January 21, 2021, 4:55pm UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/4 "2021-01-21T16:55:53Z")
> [@ced](#):
>
> Indeed instead of using `timeit`, `pyperf` should be used.
In fact `pyperf` is just a wrapper around `timeit` to provide some statistics. But it provides the nice `pyperf system tune` command that can be used to set IRQ affinity and so on (I should reboot in order to isolate a CPU afterwards to test that as well).
> [@JCavallo](#):
>
> From working on “performances” related problem rather frequently, I think there may be a misinterpretation, probably caused by CPU throttling / other processes.
To be honest, the results here are consistent with those I had yesterday (I ran those in the console without running anything else not even the cronjobs). But as I did not have them in a file, I just ran the test again and checked that they where consistents.
Now that I have use `pyperf system tune` here are the results:
- C implementation: 2.10175000000163
- Pure python implementation: 1.5666235190001316
And indeed the timing of the calls in the profiler are more alike.
I know it seems strange, as I said one of the reason for the difference is to search in the way I implemented the accelerator maybe:
```python
try:
from ._field import Field as CField
except ImportError:
class CField(object):
def __get__ (self, inst, cls):
if inst is None:
return self
assert self.name is not None
if self.name == 'id':
return inst._id
return inst. __getattr__ (self.name)
```
And then `Field` inherits from `CField` (that way we can fallback on a pure Python implementation if the accelerator is not present). Maybe that this additional level of indirection is what slowly builds up to this difference.
---
### Author:  [@rmu](https://discuss.tryton.org/u/rmu)
#### Post date: [January 22, 2021, 9:41am UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/5 "2021-01-22T09:41:40Z")
Did you try running the benchmark with the pypy jit?
---
### Author:  [@albert](https://discuss.tryton.org/u/albert)
#### Post date: [January 22, 2021, 9:50am UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/6 "2021-01-22T09:50:27Z")
> [@nicoe](#):
>
> The implementation of ` __get__ ` is quite naive, but it’s only for testing purpose. But anyway given the dynamicity of Python I think that the only part where some improvements could be done would be in the way we handle the `name == 'id'` case (by fetching directly the data from the object’s struct).
You could try leaving \_\_get\_\_ implementation in “class Field” as:
```
def __get__ (self, inst, cls):
if inst is None:
return self
return ins. __getattr__ (self.name)
```
and add a new class FieldId with its own \_\_get\_\_ implementation:
```
def __get__ (self, inst, cls):
if inst is None:
return self
return inst._id
```
---
### Author:  [@albert](https://discuss.tryton.org/u/albert)
#### Post date: [January 22, 2021, 9:58am UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/7 "2021-01-22T09:58:13Z")
> [@nicoe](#):
>
> The first thing to do is to benchmark what you’re working on. In my case it concerns access to Tryton fields. I used a small script with `trytond-console` in order to create 200 000 move lines.
Also, I would compare how much it takes running the script with just a 1000 moves (that is, without going beyond the cache boundary).
This is important in order to understand what is the performance of the methods involved in just the plain access to fields once they’re in memory and what is the performance when data must be loaded.
In the latter case, you may simply increase record cache size to say 10000 and see its impact on the performance.
---
### Author:  [@albert](https://discuss.tryton.org/u/albert)
#### Post date: [January 23, 2021, 4:35pm UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/8 "2021-01-23T16:35:20Z")
I also suggest you take a deep look at this:
> [@nicoe](#):
>
> ` 1470000 5.041 0.000 22.344 0.000 /home/nicoe/projets/tryton/tryton-env/tryton-env/trytond/trytond/model/modelstorage.py:1555(instantiate)`
You’re just looping over 200.000 records but you get 1.47M records instantiated (7 times the number of records you need) which is basically the same I [already reported](https://discuss.tryton.org/t/should-create-uid-write-uid-use-lazy-loading/2458).
You can improve that situation a lot by:
- Lazy loading create\_uid/write\_uid
- Lazy loading all M2o fields
I would also do some checks with:
- Lazy loading write\_date, create\_date
- Lazy loading all the fields that you’re not using in the loop
---
### Author:  [@albert](https://discuss.tryton.org/u/albert)
#### Post date: [January 23, 2021, 4:44pm UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/9 "2021-01-23T16:44:38Z")
> [@nicoe](#):
>
> A good candidate would be our cache implementations as they could be quite independent of the rest and yet the do appear at the top of the methods that are slow (not THAT slow but a bit slow anyway).
Mabye you can make a quick comparison with this C implementation of LRU Dict:
> **[lru-dict](https://pypi.org/project/lru-dict/)**
>
> An Dict like LRU container.
That said, it seems that we spend a lot of time in \_check\_size\_limit. Probably because once the cache is filled-in with each new record the oldest one must be removed. Maybe it would be better if the cache was more “flexible”. So instead of having a hard limit (which is 1000 by default) there was a soft and hard limit. So, once the hard limit is reached “hard - soft” records are removed from the cache at once. Say hard = 1500 records, and soft = 1000. Then, once we reach 1500 records, the 500 least recently used are removed at once.
This idea supposes that it’s faster to remove 500 records at once rather than one by one, but I’m not sure that is true. Just an idea for a possible improvement.
Another possibility would be to make record\_cache\_size dynamic. So, by default it is 100, but once the first 100 records have been loaded, for that record list, then it becomes 1000, and after we’ve read those 1000 records, then it is increased to 10000. So, the larger the number of records we read, the larger the number of records we read at once (with upper limits of course).
---
### Author:  [@ced](https://discuss.tryton.org/u/ced)
#### Post date: [January 23, 2021, 9:45pm UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/10 "2021-01-23T21:45:36Z")
> [@albert](#):
>
> Mabye you can make a quick comparison with this C implementation of LRU Dict:
I doubt there will be any improvement against an optimized version of LRUDict.
By optimized I mean that it should be possible now to make an implementation using [only dict and not OrderedDict](https://bugs.tryton.org/issue10025) which adds a doubly linked list.
> [@albert](#):
>
> This idea supposes that it’s faster to remove 500 records at once rather than one by one, but I’m not sure that is true.
There is no method to remove more than one record at once.
> [@albert](#):
>
> Another possibility would be to make record\_cache\_size dynamic. So, by default it is 100, but once the first 100 records have been loaded, for that record list, then it becomes 1000, and after we’ve read those 1000 records, then it is increased to 10000. So, the larger the number of records we read, the larger the number of records we read at once (with upper limits of course).
And so there is no limit and all memory is used for cache.
---
### Author:  [@albert](https://discuss.tryton.org/u/albert)
#### Post date: [January 24, 2021, 12:24am UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/11 "2021-01-24T00:24:56Z")
> [@ced](#):
>
> And so there is no limit and all memory is used for cache.
There’d be a limit as I already stated. You could set that limit to 10000 or whatever is defined. The speed at which the cache\_size increases could also be defined.
The problem with the current implementation is that say that you have a loop like this:
```
for invoice in invoices:
if invoice.lines:
break
```
if invoices have a large number of lines, they will all be loaded even if it’s never necessary. So if cache\_size is set to a large number because you need to process 200 thousand invoices (just in Nicolas example) you’re loading lots of lines even if you only need a large cache size for invoices (not for the lines).
So if preloading starts with 10 records, later jumps to 100, later 1000, and eventually 10000 (and stay at 10000 records from there on if 10000 has been set as the upper limit), we will only be using a lot of cache memory for those models and lists that really use it.
Seeing the results of [lru-cache](https://bugs.tryton.org/issue10025#msg64059) I think that is the way to go but another approach may be to clear the whole cache when [read() is going to completely fill it](https://hg.tryton.org/trytond/file/tip/trytond/model/modelsql.py#l817)
Something in the lines:
```
if len(result) >= cache.size_limit:
cache.clear()
```
This should prevent the removal of item by item which is probably going to be slow. Would only apply if records are not already in the cache, so some extra logic would be required.
---
### Author:  [@ced](https://discuss.tryton.org/u/ced)
#### Post date: [January 24, 2021, 1:17am UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/12 "2021-01-24T01:17:30Z")
> [@albert](#):
>
> if invoices have a large number of lines, they will all be loaded even if it’s never necessary.
That’s always the risk of pre-loading data. But it can always be tune by the developer.
> [@albert](#):
>
> So if preloading starts with 10 records, later jumps to 100, later 1000, and eventually 10000 (and stay at 10000 records from there on if 10000 has been set as the upper limit), we will only be using a lot of cache memory for those models and lists that really use it.
This is not how the code works. We do not fill the cache but we read per `IN_MAX`.
> [@albert](#):
>
> Seeing the results of [lru-cache](https://bugs.tryton.org/issue10025#msg64059) I think that is the way to go
lru-cache can not be used because it is missing default\_factory.
> [@albert](#):
>
> but another approach may be to clear the whole cache when [read() is going to completely fill it](https://hg.tryton.org/trytond/file/tip/trytond/model/modelsql.py#l817)
Sorry but it makes no sense to me. It is not the job of the caller to worry about the internal of the container.
Also normally we always read less than the size of the cache. So your case will never happen.
> [@albert](#):
>
> This should prevent the removal of item by item which is probably going to be slow. Would only apply if records are not already in the cache, so some extra logic would be required.
It could be a way to implement LRUDict that when reaching the limit, it clear itself (of course before adding the new value).
But clearing a dict is still O(n) (as it has to loop over all keys) but deleting a key is average O(1) (worst O(n)). So in the best case it has the same complexity and the worst it is twice faster. But this is without counting the benefit that former cache value may be reused for example when there are duplicate ids in a list or if records has the same target etc.
---
### Author:  [@nicoe](https://discuss.tryton.org/u/nicoe)
#### Post date: [January 25, 2021, 9:59am UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/13 "2021-01-25T09:59:41Z")
> [@albert](#):
>
> Maybe it would be better if the cache was more “flexible”. So instead of having a hard limit (which is 1000 by default) there was a soft and hard limit. So, once the hard limit is reached “hard - soft” records are removed from the cache at once. Say hard = 1500 records, and soft = 1000. Then, once we reach 1500 records, the 500 least recently used are removed at once.
When I said that it’s where I think we could see improvements it’s indeed the kind of idea I had in mind but it was just an idea.
> [@ced](#):
>
> There is no method to remove more than one record at once.
In case we implement our own cache nothing prevent us from creating one (apart the fact that it’s an added source of bug and so on obviously).
But I agree wholeheartedly with the idea of first testing with the new constraint on the order of the keys that brings python 3.7 (although I like the fact that OrderedDict conveys the semantic way better and that I find this constraint a bit strange but this is more a philosophical objection 😃).
> [@rmu](#):
>
> Did you try running the benchmark with the pypy jit?
Hello back @rmu, no I didn’t try using pypi because the goal is the find a better implementation for Tryton not to test if another interpreter would be better 🙂. But you’re probably right I guess that the JIT would bring some improvements in this area without any dev …
---
### Author:  [@numaelis](https://discuss.tryton.org/u/numaelis)
#### Post date: [April 15, 2024, 1:49am UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/14 "2024-04-15T01:49:59Z")
Hi!
using some sort of load cache before fetching the table values?
like this:
```auto
suma = Decimal(0)
for move in move_lines:
move.select_cache(["debit","credit"],'account_move_line')
res = move.debit - move.credit
suma+=res
```
```auto
def select_cache(self, list_names, table):
if self.id:
cur = Transaction().connection.cursor()
field_str = ",".join(list_names)
cur.execute("SELECT "+ field_str+" FROM "+table+" where id="+str(self.id)+" ;")
result = cur.fetchone()
if result:
ix = 0
for name in list_names:
if self._local_cache.get(self.id,-1)==-1:
self._local_cache[self.id] = {}
self._local_cache[self.id][name] = result[ix]
ix+=1
```
---
### Author:  [@numaelis](https://discuss.tryton.org/u/numaelis)
#### Post date: [April 15, 2024, 1:53am UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/15 "2024-04-15T01:53:59Z")
using cprofile:
```auto
suma = sum(l.debit - l.credit for l in move_lines) # noqa
```
27421 function calls (25756 primitive calls) in 1.575 seconds
```auto
suma = Decimal(0)
for move in move_lines:
move.select_cache(["debit","credit"],'account_move_line')
res = move.debit - move.credit
suma+=res
```
27421 function calls (25756 primitive calls) in 0.632 seconds
Maybe this is useful when there are many records and we want to maintain some pythonic style
---
### Author:  [@ced](https://discuss.tryton.org/u/ced)
#### Post date: [April 15, 2024, 7:11am UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/16 "2024-04-15T07:11:35Z")
What are the condition of such test?
Because I doubt that making a SQL query at each iteration would be faster than the current preload of cache per bunch.
Is the list of `move_lines` has the cache correctly align?
How much records are there?
---
### Author:  [@numaelis](https://discuss.tryton.org/u/numaelis)
#### Post date: [April 15, 2024, 12:12pm UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/17 "2024-04-15T12:12:56Z")
The previous test was in an old version of tryton.
Now I have another test for version 6.4
account\_move\_line table has 1609 rows
wearing:
sum = sum(l.debit - l.credit for l in move\_lines) # noqa
I obtained(adjusting cprofile):
```auto
892669 function calls (796329 primitive calls) in 0.466 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
76683/3238 0.016 0.000 0.385 0.000 field.py:398( __get__ )
6449/3231 0.063 0.000 0.383 0.000 modelstorage.py:1622( __getattr__ )
```
although sometimes the time was considerably reduced to this:
```auto
104045 function calls (87219 primitive calls) in 0.122 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
2/1 0.001 0.000 0.093 0.093 modelsql.py:1744(search)
5 0.000 0.000 0.059 0.012 database.py:65(execute)
```
using select before
```auto
105657 function calls (95267 primitive calls) in 0.273 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1614 0.002 0.000 0.183 0.000 database.py:65(execute)
1614 0.178 0.000 0.181 0.000 {function LoggingCursor.execute at 0x7f2870ff6a60}
1609 0.013 0.000 0.150 0.000 modelstorage.py:1601(select_cache)
```
---
### Author:  [@ced](https://discuss.tryton.org/u/ced)
#### Post date: [April 15, 2024, 1:15pm UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/18 "2024-04-15T13:15:21Z")
I do not understand what I’m reading.
---
### Author:  [@numaelis](https://discuss.tryton.org/u/numaelis)
#### Post date: [April 15, 2024, 2:03pm UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/19 "2024-04-15T14:03:56Z")
ok. what I understand happens in:
```auto
sum = sum(l.debit - l.credit for l in move_lines) # noqa
```
is that when doing l.debit, somewhere in the code it is a select debit from table
and when doing l.credit, it is another select credit from table
there are two select.
but if the select is reduced, it will logically be more efficient when you want to traverse a table. That’s why we use classmethod function fields with pysql. but sometimes it is more comfortable (practical) to use the orm, like l.debit - l.credit. which will not cause problems with few rows.
here I simply wanted to speed up the example:
sum = sum(l.debit - l.credit for l in move\_lines) # noqa
but I don’t know if it would be good to add an annotation to modelsql before going through large data.
---
### Author:  [@numaelis](https://discuss.tryton.org/u/numaelis)
#### Post date: [April 15, 2024, 2:09pm UTC](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715/20 "2024-04-15T14:09:56Z")
The cprofile code I used was this.
I used it temporarily in the invoices module, and from the client I went to invoices, I clicked on the search button
```auto
@classmethod
def read(cls, ids, fields_names):
result = super(Invoice, cls).read(ids, fields_names)
pr = cProfile.Profile()
pr.enable()
MoveLine = Pool().get('account.move.line') # noqa
move_lines = MoveLine.search([])
#suma = sum(l.debit - l.credit for l in move_lines) # noqa
suma = Decimal(0)
for move in move_lines:
move.select_cache(["debit","credit"],'account_move_line')
res = move.debit - move.credit
suma+=res
pr.disable()
out_stream = open('estadis.txt', 'w')
ps = pstats.Stats(pr, stream=out_stream)
ps.strip_dirs().sort_stats('cumulative').print_stats()
return result
```
[Next page](https://discuss.tryton.org/t/using-c-extensions-to-speed-up-tryton/3715.md?page=2)