Using JS libraries inside a multithreading environment
Imagine you want to use external libraries like AmCharts, Google Analytics or MapboxGL inside a webworkers driven environment, where your application logic is located inside a webworker. This can be a problem!
For neo.mjs, this is exactly the case: most parts of the framework as well as the apps you create run inside the App worker.
In case you have not looked at neo.mjs yet, you should:
The framework is based on top of ES8 and the dev mode can run inside Google Chrome (v80+) without the need for any JS builds at all. With a custom JSON based virtual DOM engine in place (located in another worker), the performance to update the real DOM is incredibly fast and you can easily craft beautiful apps like this:
Inside the second screenshot you can already see that MapboxGL is in use, so how does this work?
neo.mjs just got enhanced with a new main thread addons system. The goal was to enable you to pick which addons you want to use and only load those into the main thread to keep main as lightweight as possible. It is also now possible to create new addons very easily, so you are welcome to contribute on this part!
You can take a look at the current addons here:
https://github.com/neomjs/neo/tree/dev/src/main/addon
After following the getting started guide:
https://github.com/neomjs/neo/blob/dev/.github/GETTING_STARTED.md
you can easily generate a new app like this:
> npm run create-app
The script will give you 3 questions:
The 3rd one is new and allows you to pick your addons. No worries, you can change them afterwards.
Looking at the generated index.html file:
You will see your selected mainThreadAddons right away. This is also the spot where you want to change them for the dev mode.
Since we already noticed that MapboxGL is in use, let us take a look at the addon:
https://github.com/neomjs/neo/blob/dev/src/main/addon/MapboxGL.mjs
Now you might be wondering why we load the built scripts of the library and are not importing the related node_module. Truth is: I would love to do this, but the neo.mjs dev mode has to run without any JS builds at all. The problem here is that most npm packages use invalid import statements, which are missing the file name extensions. A browser on its own has no chance to understand these. This starts with Typescript generating “wrong” imports already. Chances are pretty good that this will change at some point, since not using file name extensions does not follow the import specs.
Back to our demo app: Inside the 2nd screenshot there are map controls. You can toggle the visibility of the heat map or the detail circles on or off:
https://github.com/neomjs/neo/blob/dev/apps/covid/view/mapboxGl/ContainerController.mjs#L26
This logic happens inside the app worker. We are calling setLayoutProperty() on the map component. This one is defined here:
https://github.com/neomjs/neo/blob/dev/src/component/wrapper/MapboxGL.mjs#L342
Now you should be confused!
Neo.main.addon.* is located inside the main thread, so the class (singleton) does not exist inside the app worker. So, how can we call a method directly which should not exist inside the app worker scope?
This is where the power of the remotes API kicks in.
The method is obviously defined inside the main thread:
https://github.com/neomjs/neo/blob/dev/src/main/addon/MapboxGL.mjs#L369
Defining an addon method alone won’t make it available inside another worker, since we do not want to expose all methods.
Exposing the methods you want to share with other workers is trivial though:
Just drop them into the remote config for the target worker (app in most cases) and as soon as you do, you can call them as a promise inside the app thread directly. Under the hood app is sending a post message to the main thread, the method gets called, main is sending a response to app and you can use then() to get return values.
Since the neo MapboxGL wrapper component is directly connected to the matching addon, you can easily define map related properties via JSON inside the app worker:
https://github.com/neomjs/neo/blob/dev/apps/covid/view/mapboxGl/Component.mjs
You might be wondering as well how optional addons can work directly inside the browser as well as inside a webpacked based dist environment. You will find the answer here:
https://github.com/neomjs/neo/blob/dev/src/Main.mjs#L136
The clue here is line 151: webpack has no chance to know which files could be inside the addon folder, so it does grab every file, which is exactly what we want for the build process. When opening your dist version app, webpack will only load the required addon chunks. I scoped addons into a main thread folder to ensure there are no chunk conflicts with other workers.
Have fun using the default neo.mjs main thread addons or add new ones as needed to the repository (PRs are welcome!).
Addons are intended to load external libraries and create wrapper functions to work with APIs of other libraries. They are NOT intended to manipulate the app related DOM (for this you have the vdom engine in place).
Feedback is appreciated and feel free to jump into the neo slack channel in case you have questions! The invite link is at the bottom of the repo readme.
In case you did already clone the neo.mjs repo, i recommend to delete the dist folder after doing a pull and run the buildAll script.
Best regards & happy coding,
Tobias
