7 Questions to Ask When Designing and Implementing Background Jobs

How to Design and Implement Robust Background Jobs in Our Applications

Most background jobs are running behind the scene and are transparent to end-users, and admins usually don’t need to worry about the nitty-gritty of how the system automates tasks or when the system executes them. In some cases, product owners may know that there are background jobs supporting the application because they are involved in the requirement gathering phase of the application development lifecycle. However, as time passes or initial product owners transit away, corporate knowledge of background jobs starts to fade, unless the current product owners are tech-savvy users or they have detailed documentation at hand. Sadly, the unawareness of background jobs is a natural ending in situations like this.

When people start forgetting things, the application begins to behave counter-intuitively. Sooner or later, we will figure out that something is broken. The malfunctioning could be caused by many reasons, such as a server maintenance window, edge cases in the input data, a new business process without being informed and programmed, and so on. Even worse, admins start to hack the system to make it “work”, which could further mess up the system state.

As developers, we want to avoid patching our applications to simply hotfix problems. Instead, we should design the system in a way that the background jobs are fault-tolerant and friendly to newcomers.

In this article, we will review some key questions to ask ourselves when we design and implement background jobs. In our discussion, we will focus on recurring jobs that run repetitively at certain schedules which are usually programmed using CRON expressions. The following key questions may also apply to other system design processes.

Does the Background Job Properly Handle Upstream Failures?

A background job is similar to a black box that processes input data and emits final results. The input data could be data…