Feasability to replace sao (js) by python for web client

[risto3]

@edbo

The 162M for sao is the development version. You have to check the production-version which should not be bigger than 5M.

according to npm help install

With the --production flag (or when the NODE_ENV environment variable
is set to production), npm will not install modules listed in
devDependencies.

What makes you think the development versions got installed?

It’s even bigger if I do the audit fix:

$ npm audit fix

phantomjs-prebuilt@2.1.16 install /home/richard/src/tryton-env/sao/node_modules/phantomjs-prebuilt
node install.js

Considering PhantomJS found at /usr/sbin/phantomjs
Found PhantomJS at /usr/sbin/phantomjs …verifying
Error verifying phantomjs, continuing { Error: Command failed: /usr/sbin/phantomjs --version
/usr/sbin/phantomjs: error while loading shared libraries: libicui18n.so.62: cannot open shared object file: No such file or directory

at ChildProcess.exithandler (child_process.js:294:12)
at ChildProcess.emit (events.js:189:13)
at maybeClose (internal/child_process.js:978:16)
at Socket.stream.socket.on (internal/child_process.js:395:11)
at Socket.emit (events.js:189:13)
at Pipe._handle.close (net.js:610:12)
killed: false,
code: 127,
signal: null,
cmd: ‘/usr/sbin/phantomjs --version’ }
Downloading https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2
Saving to /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
Receiving…
[=======================================-] 99%
Received 22866K total.
Extracting tar contents (via spawned process)
Removing /home/richard/src/tryton-env/sao/node_modules/phantomjs-prebuilt/lib/phantom
Copying extracted folder /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1545824881689/phantomjs-2.1.1-linux-x86_64 → /home/richard/src/tryton-env/sao/node_modules/phantomjs-prebuilt/lib/phantom
Writing location.js file
Done. Phantomjs binary available at /home/richard/src/tryton-env/sao/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs
added 337 packages from 223 contributors and updated 1 package in 27.857s
fixed 4 of 65 vulnerabilities in 1380 scanned packages
1 vulnerability required manual review and could not be updated
7 package updates for 60 vulns involved breaking changes
(use npm audit fix --force to install breaking changes; or refer to npm audit for steps to fix these manually)
$ du -hs .
259M .

Regardless, it is a mess and seemingly highly vulnerable… more than questionable for a production machine.

[nicoe (Nicolas Évrard)]

@risto3

What makes you think the development versions got installed?

You counted everything, sao is way lighter than that.

nicoe@mirabelle:~/projets/tryton/…/tryton-env/sao% du -h dist
1,3M    dist

And dist contains both the minified and the vanilla version.

It does not change that the javascript world is a mess but we have to do with what is available

[semarie (Sebastien Marie)]

@risto3

What makes you think the development versions got installed?

npm stuff is a pain to package properly (aka having only minimal number of files required).

you could check if you have any bower_components/*/docs directories. for example, papaparse/docs is ~50Mo here.

@risto3

Regardless, it is a mess and seemingly highly vulnerable… more than questionable for a production machine.

it isn’t so simple.

for all the packages used, only some are effectively used at runtime: others are used at “build time” on trusted source code (sao code).

and for effectively installed packages, code will run on client side to communicate with trusted code (trytond server).

[semarie (Sebastien Marie)]

@nicoe

sao is way lighter than that

I think the problem is mostly with weight of dependencies

you can’t run sao with only sao code (needs jquery, bootstrap, etc…)

[risto3]

@nicoe

You counted everything, sao is way lighter than that.

nicoe@mirabelle:~/projets/tryton/…/tryton-env/sao% du -h dist
1,3M    dist

And dist contains both the minified and the vanilla version.

It does not change that the javascript world is a mess but we have to do with what is available

Well, perhaps in more detail:

$ du -hs * |sort -h
9,0K bower.json
9,0K CHANGELOG
9,0K COPYRIGHT
9,0K Gruntfile.js
9,0K index.html
9,0K package.json
9,0K README
25K LICENSE
36K tests
65K package-lock.json
368K locale
440K src
493K dist
530K images
40M bower_components
116M node_modules

where it seems dist/, node_modules/ and bower_components/ are generated via npm & grunt

[risto3]

@semarie

you could check if you have any bower_components/*/docs directories. for example, papaparse/docs is ~50Mo here.

===================

$ find . -name docs -exec du -hs {} ; | sort -h
19K ./node_modules/bower/lib/node_modules/handlebars/docs
1,2M ./bower_components/eonasdan-bootstrap-datetimepicker/docs
2,5M ./bower_components/c3/docs
23M ./bower_components/papaparse/docs

[risto3]

Let me ask perhaps a real stupid question, how did OpenERP do their web client without all the node.js hocus pocus and bloat; I believe it was also in javascript, no?

[semarie (Sebastien Marie)]

@risto3

40M bower_components
116M node_modules

node_modules aren’t necessary for deployment (not need on the webserver).

as you noted, bower_modules contains documentation that aren’t necessary for deployment.

on OpenBSD, the port for sao install a directory with 15.5Mo (well, currently it is 67Mo, but your post makes me consider to check, and I am cooking a diff to remove docs/ directories ).

[nicoe (Nicolas Évrard)]

@semarie

you can’t run sao with only sao code (needs jquery, bootstrap, etc…)

Indeed but it’s a bit like saying “you can not run tryton without gtk”.
jQuery + bootstrap is a bit like gtk (but we have to build everything from scratch), etc.

Of course it’s like having one gtk library per browser tab but that’s the way the web works …

@risto3

Let me ask perhaps a real stupid question, how did OpenERP do their web client without all the node.js hocus pocus and bloat; I believe it was also in javascript, no?

Well if you go on their github page you’ll see that the project is 46% Javascript and 43.4% python, so yes they have a lot of javascript. And I think that their web client is distributed through their web addons:

And they use grunt, npm, etc to build their client.

[risto3]

If so, then they probably use it upstream and statically deploy the .js… I see this on our current 6.1 server:

src/odoo/addons$ du -hs web* |sort -h
70K web_rpc
89K web_tests
112K web_hello
359K web_uservoice
394K web_livechat
521K web_kanban
522K web_dashboard
623K web_process
696K web_diagram
803K web_gantt
1,3M web_graph
2,6M web_mobile
2,7M web_calendar
7,0M web

No idea for 7.0+ as we abandoned ship like many others staying on 6.1 until migration.

[updated] it appears so as I see the following:

src/odoo/addons$ find web/static/ -maxdepth 2 -type d
web/static/
web/static/test
web/static/src
web/static/src/img
web/static/src/css
web/static/src/xml
web/static/src/js
web/static/lib
web/static/lib/jquery.deferred-queue
web/static/lib/jquery.scrollTo
web/static/lib/jquery.superfish
web/static/lib/jquery.validate
web/static/lib/jquery.ui.timepicker
web/static/lib/json
web/static/lib/jquery.contextmenu
web/static/lib/jquery.form
web/static/lib/jquery.blockUI
web/static/lib/qunit
web/static/lib/jquery.tipsy
web/static/lib/labjs
web/static/lib/jquery.ui.notify
web/static/lib/py.parse
web/static/lib/jquery.ui
web/static/lib/qweb
web/static/lib/jquery.MD5
web/static/lib/jquery.ba-bbq
web/static/lib/underscore
web/static/lib/datejs
web/static/lib/jquery

[nicoe (Nicolas Évrard)]

@risto3

If so, then they probably use it upstream and statically deploy the .js

For the deployment, you’re probably right but you can do the same with sao.

[risto3]

What I mean by upstream ‘deploy’ is what is committed into the DVCS (git, in this case).

[nicoe (Nicolas Évrard)]

Then why would there be a Gruntfile.js in their source ?
Also it’s a bit stupid to not put in your DVCS the source of your building process.

[risto3]

Beats me… just never needed to do anything to the upstream web sources like npm/grunt/etc.

odoo/odoo

But, naturally, the source is there - otherwise how could one do an in-tree update?

BTW - I don’t find Gruntfile.js… perhaps that came in a later version…

[nicoe (Nicolas Évrard)]

@risto3

Beats me… just never needed to do anything to the upstream web sources like npm/grunt/etc.

You’re talking about an old version of odoo (I don’t know exactly how old it is) … now they have switched to better practice it seems.

@risto3

BTW - I don’t find Gruntfile.js … perhaps that came in a later version…

The link I provided has it. They’re using a more standard methodology instead of vendoring the libs like they used to do.

[plaes (Priit Laes)]

A while ago I actually tried to address this by creating a build-target to create distributable zip file for deploying sao. Unfortunately I was met again with resistance…

But patch is still available here: Issue 7046: Add build-zip task for creating distributable archive - Tryton issue tracker

[semarie (Sebastien Marie)]

@nicoe

The link I provided has it. They’re using a more standard methodology instead of vendoring the libs like they used to do.

for me, they are still vendoring external libraries: odoo/tree/12.0/addons/web/static/lib

[fmorato (Felipe Morato)]

@edbo

That’s quite interesting indeed. But you have to learn a new language and it used JVM which can also be a pain in the back because of different versions.

I’m not sure what you mean about the JVM. Of course there is a new language involved but it doesn’t require JVM to run.

https://kotlinlang.org/docs/reference/native-overview.html