Circuit Breakers: The Saviour of your Microservices
The world has moved towards a distributed environment containing lots of microservices. The benefits of microservices architecture are abundant but that doesn’t mean that they are without failures. Rather, failure of any service is inevitable. When service(s) are unavailable or taking too long to respond, this can result in a cascading failure over the chain of service calls (which could scale up all the services to handle the traffic or bring it down completely).
When one service invokes another service, there are chances that the other service is unavailable or taking too long to respond, which leads to valuable resources like threads being engaged in waiting for the service to respond. A consequence of this could be resource exhaustion and the service being called will fail to handle incoming requests. The circuit breaker pattern is all about preventing such catastrophic events and adds to building a fault-tolerant and resilient system that could survive such events comfortably, without being unavailable or auto-scaling unnecessarily (incurring extra cost).
Circuit Breaker Pattern
As with an electrical circuit breaker, when the service encounters a number of failures, the circuit breaker trips for a particular duration. In that period, all the attempts made to invoke the failing service will fail immediately. Once the configured duration is over, the circuit breaker will allow a certain number of requests to pass through and monitor whether the requests succeed (in this case, the circuit breaker resumes the service as normal). If there are failure(s) then the circuit breaker continues to be tripped for a given period.
Therefore the client that invokes the service should actually be wrapped through a proxy, which monitors for failures. When it encounters failures, it starts to function like a circuit breaker. Hence, this pattern allows the service to handle such failures gracefully. The circuit breaker has the following states:
CLOSED
The circuit breaker being in a CLOSED state means that everything is working fine and all calls pass through to the remote services. Once the number of failures exceeds a predetermined threshold, the circuit breaker trips and enters into the open state.
OPEN
Once the number of timeouts reaches a predetermined threshold in the circuit breaker, it trips the circuit breaker to the OPEN state. In the OPEN state, the circuit breaker returns an error for all calls to the service without making the calls to the remote service.
HALF-OPEN
After a certain duration, the circuit switches to a HALF-OPEN state to test if the underlying problem still exists. The circuit breaker uses a mechanism to make a trial call to the remote service periodically to check if it has recovered. If the call to the Remote service fails, the circuit breaker remains in the OPEN state. If the call returns success, then the circuit switches to the CLOSED state. The circuit breaker then returns all external calls to the service with an error during the HALF-OPEN state.
Implementation
The two most popular libraries for fault tolerance in JAVA are
Example Circuit Breaker using Spring boot
If you are using Spring Boot, you can use the circuit breaker implementation from Netflix Hystrix fault tolerance library.
The EnableCircuitBreaker annotation tells Spring Cloud that the application uses circuit breakers that will enable their monitoring, opening, and closing as per availability of tracking service.
The HystricCommand initiates the initial configuration, if the circuit breaker trips then it needs to call defaultProductList method. Also, we have specified our timeout period through which the circuit breaker determines if the call is a failure or not.
Conclusion
Circuit breakers are also very valuable place for monitoring. Whenever the state of the circuit breaker changes, it should be logged and reveal details of their state for deeper monitoring. Circuit Breaker’s behavior is often a good source of warnings about deeper troubles in the environment. Operations staff should also be able to trip or reset breakers. Thanks for reading, I hope you found this article to be helpful!
