“data-manager” — A single source of truth for real-time JavaScript Applications

In my few years of experience in JavaScript development, I have discovered that with increasing application complexity, the biggest challenge is efficiently providing “up-to-date” data to each component. In real time applications — such as IoT — this challenge is potentiated by the fact that data is constantly being updated asynchronously, through a bidirectional connection with the application back-end.

To simplify and centralise the management of application data, during the years of web application development ranging from jQuery, AngularJS and finally Angular, I have established a data-manager service pattern — a single and always up-to-date source of truth for the entire JavaScript client application.

First, let me share some background how our applications are structured. Back-end provides a HTTP REST API for retrieving data and submitting user actions, as well as a web-socket server for bi-directional communication with the client application. Client application works with data in the following way:

  • It initially retrieves a complete state (data set) for the current user, consisting of several aggregate root objects (or entities) with several sequential or parallel HTTP GET requests to the REST API.
  • Along with loading the data for each aggregate root, client application subscribes to the web-socket server with unique aggregate root id in order to be notified about any future changes in the given aggregate root state.
  • As a result of user interaction, client application invokes asynchronous HTTP POST/PUT operations on the REST API. Successful HTTP response only signals that command has been successfully received by the back-end.
  • Once command has been actually processed, confirmation or rejection is passed back to the invoking client via a web-socket event.
  • Background changes in the state of the aggregate roots (either from other clients or from actual devices) are propagated to the application via web-socket events.

To better illustrate the situation, here is a simplified example from SeltronHome platform. When user logs in to the client app, client app first retrieves e a list of subscriptions he can access (he can either be subscription owner or a guest). Next, details for each subscription, including subscription owner and resource groups — buildings, must be retrieved. Each building has one or more GWD devices — resources — and each GWD can have a Seltron Heating Regulator — connected controller — attached to it.

SeltronHome data model structure

For instance, room temperature component in Clausius displays currently measured temperature in the room and enables user to adjust preferred temperature. It initially requires current state of the heating regulator and has to update immediately when measured or set-point temperature has changed on the heating regulator or in another instance of the client application.

Clausius room temperature component

The room temperature component must first retrieve current state of the heating controller from the API, subscribe to the web-socket server to receive any state updates and register handlers for set-point and measured temperature changed events. When user changes the set-point temperature, it must invoke a HTTP PUT request on the API and wait for the command result event on the web-socket server. In case the command fails, the previous state of the heating controller must be restored.

Room temperature component communication flow

For a simple component this communication flow might not seem a big deal, but if several components access the same heating controller state, the whole pattern is repeated over and over again.

To overcome the above presented challenges, I use the data-manager service, which is responsible for retrieving, maintaining and serving the one single data object of the application state. Upon application initialisation, data-manager loads entire application data set into an independent data object. It subscribes to any available web-socket events in order to ensure prompt updates when data changes in the back-end.

Component communication with data-manager

Instead of working with several HTTP services, each component retrieves a reference to the main data object or its part from the data-manager. When data-manager updates any field in the fore-mentioned data object, either based on the user action in another component or a web-socket event, the changes are immediately reflected in data model of every component. With AngularJS or Angular change detection mechanisms, a single notification to update ($rootScope.$apply or NgZone.run respectively) suffices to refresh the views of every application component displaying information in question. In case a component requires any specific processing of the changed data, data-manager can also expose an observable that enables the component to react with custom behaviour when the data has changed.

To optimise the loading and response times, data-manager also works as a flexible in-memory cache solution. With regular HTTP cache, you would have to invalidate and reload cached requests each time a web-socket update is received. On the other hand, data-manager can update the affected fields in the data model with values provided in the event payload, thus removing the need for additional HTTP requests to the API. Furthermore, for rarely changing data, such as translations or heating controllers specifications (objects, describing device capabilities based on it’s model or serial number), data-manager also enables centralised storing to and reading from the browser local storage.

If required, data-manager could also easily store the entire data object into the local storage and read it when the application starts. This way application can immediately display the full view (with out-of-date data of course) and then gradually update the data in the background and update the displayed view.

The above solution has proven to work well in my applications and I personally feel that it is easily understandable and manageable. The main advantage is in centralised data management with straightforward asynchronous updates and caching, while the biggest disadvantage is initial loading time and disability to separate concerns to smaller parts of data — entire data object (tree) must be loaded in advance. I am aware there are other concepts, patterns and libraries that are trying to solve the same problem. I do not have enough practical experience with the rest to make a quality comparison. I have done some research about React and Redux data architecture, but I have not yet used that in a production application. Recently this RxJS state management library for Angular has also been brought to my attention and I intend to explore it in the near future.

I would really value any other opinion or discussion about pros and cons of each approach. Please don’t hesitate to share yours!

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.