Web-Worker RPC with RabbitMQ

Node.js example of web-worker remote procedure call (RPC) with RabbitMQ.

A Use Case

You have a Node.js web application that is required to upload images and return thumbnails of them.

One option is to simply use a cloud service, e.g., Cloudinary; but you are concerned about long-term costs.

You find that the Node.js sharp library has all the image processing features you need (and more).

Knowing that Node.js is single-threaded, you know that you cannot run blocking compute intensive operations in the web application code; you will need a worker process.

Alert: After I already wrote my example code, I read the following:

Everything remains non-blocking thanks to libuv, no child processes are spawned and Promises/async/await are supported.

— Sharp — Sharp

Dang it. Looks like image processing is not a good use case for a worker process. Either way, let us imagine we have an operation that is blocking compute intensive.

Node.js Web-Worker RPC Solutions

Previously I wrote a similar article, Worker Processes with Heroku By Example, using Redis and the kue Node.js library. The problem, however, is that shortly after I wrote that article the kue library was deprecated. At that time, there were no obvious replacement libraries that worked with Redis and so I fell back to using the tried and true RabbitMQ with the amqplib Node.js library.

note: As I write this article, it turns out that there is now another Redis based Node.js library, bull, that looks promising; easy to use, feature rich, and most importantly a respectable number of GitHub stars.

Why RPC is Harder with RabbitMQ

My first use case with RabbitMQ and the amqplib Node.js library did not require the web application to wait for the worker’s response; was more a fire and forget message pattern.

This pattern is particularly easy to up and running with RabbitMQ and the amqplib Node.js library; the library’s “hello world” example is pretty much all you need to get this working.

It turns out that having the web application wait for the worker’s response (a RPC or job pattern) is more challenging. The critical constraint is RabbitMQ (and thus ampqlib) is based on uni-directional communication from a publisher to a consumer using a queue; i.e., there no built-in concept of a response from the consumer back to the publisher.

note: Both of the Redis based solutions (kue and bull) support bi-directional communication; makes the RPC pattern easy when using them.

RPC with RabbitMQ

A solution for bi-directional communication with RabbitMQ is to create multiple queues as show in the following diagram; this solution is closely aligned with the RabbitMQ tutorial on this topic; RabbitMQ Tutorial: Remote Procedure Call (RPC).

1. One, of multiple, web instances publishes a message into a single queue, e.g., Task. Along with the RPC request payload, the message includes a reference to a reply queue that is unique to that web worker
2. One, of multiple, worker instances consumes the message
3. The worker instance that consumed the message then publishes a message into the supplied reply queue with a RPC response payload
4. The web instance, unique to the reply queue, consumes the message

There is one additional complication; each web instance can have multiple outstanding RPC requests. In order to disambiguate these RPC requests, the messages also carry a unique identifier, referred to as a correlation id.

The Example

The example is a web application that has a single endpoint that when browsed-to, passes the string “hello” as a RPC request payload and waits for the worker’s response. The worker responds with the supplied string joined with the word “world” as the RPC response payload. The web application, upon receiving the response, returns the joined string, i.e., “hello world” as . The solution is available for download.

Observations:

• The most complicated aspect of this solution is that the web application, src/server.ts, maintains a hash map (indexed by correlation id) of callbacks for outstanding RPC requests

Wrap Up

Nothing too fancy here; hope you found it useful.