Circuit Breaker pattern – Go implement Part 3

In the previous post, i have implemented simple circuit-breaker pattern with Java, and today i will do it with Go

Talk about advanced of Go

+ Golang has great concurrency, the memory required for creating thread is very light.

+ Code is simple and very clean, easy to lean.

+ Appropriate and effective for service application.

+ ……

You can get more information at (https://hackernoon.com/why-i-love-golang-90085898b4f7)

Focus on main issue of this post, i will talk about step by step to implement breaker pattern

+ Create package and file breaker.go into the folder

+ Define state service and state breaker

type State inttype State int
type StateService int
const (
   STATE_CLOSE  State        = 1
   STATE_HALF_OPEN     State        = 2
   STATE_OPEN          State        = 3
   SERVICE_AVAILABLE   StateService = 1
   SERVICE_UNAVAILABLE StateService = 0
)

+ Define Breaker, BreakerEvent, OptionsConfig, CounterResult

type CounterResult struct {
   TotalRequests uint32
   TotalSucceses uint32
   TotalFailures uint32
   TotalRejects uint32
}

type Breaker struct {
   options OptionsConfig
   counter CounterResult
   events []chan BreakerEvent
   state State
   consecFailures uint32
}

type BreakerEvent struct {
   Code StateService
   Message string
}

type OptionsConfig struct {
   Attempts uint32
   TimeoutState time.Duration
   LimitFailure uint32
   MaxRequests uint32
   NameService string
}

+ Create new instance breaker

func NewBreaker(options *OptionsConfig) *Breaker {
if options == nil {
   options = &OptionsConfig{}
}
if options.Attempts == 0 {
   options.Attempts = 3
}
if options.TimeoutState == 0 {
   options.TimeoutState = 4 * time.Second
}
if options.LimitFailure == 0 {
   options.LimitFailure = 5
}
if options.MaxRequests == 0 {
   options.MaxRequests = 10000000
}
if options.NameService == "" {
   options.NameService = strconv.Itoa(int(time.Now().UnixNano()))
}
return &Breaker{options: *options, counter: CounterResult{}, state: STATE_CLOSE, consecFailures: 0}
}

+ Allow other services subscribe event

// other services subcriber
func (cb *Breaker) Subcriber() <-chan BreakerEvent {
   evenReader := make(chan BreakerEvent)
   outputChannel := make(chan BreakerEvent, 100)
   go func() {
   for event := range evenReader {
      select {
      case outputChannel <- event:
      default:<-outputChannel
      outputChannel <- event
      }
   }
   }()
   cb.events = append(cb.events, evenReader)
   return outputChannel
}

+ Create execute handle

// execute handle
func (cb *Breaker) Execute(handle func() ResponseCommand, timeout time.Duration) ResponseCommand {
   // add request
   atomic.AddUint32(&cb.counter.TotalRequests, 1)
   response := ResponseCommand{}
   // Reject all invoke when current state is OPEN
   if cb.IsOpen() {
   cb.Reject()
      response.Error = cb.ErrorServiceUnavailable()
      response.Data = nil
      return response
   }
   // run handle immediate and execute time is unlimited
   if timeout == 0 {
      response = handle()
   } else {
   // run handle with time execute is limited
   c := make(chan ResponseCommand, 1)
   go func() {
      c <- handle()
   }()
   select {
      case r := <-c:
      response = r
      close(c)
      case <-time.NewTimer(timeout).C:
      response.Error = cb.ErrorTimeoutExecute()
      response.Data = nil
   }
   }
   // check for any errors
   if response.Error != nil {
      cb.Failure()
   } else {
      cb.Success()
   }
   return response
}

You can reference full source and demo at : circuit-breaker-go

Circuit Breaker pattern – Java implement Part 2

Base on part 1, today i’m going to implement circuit-breaker pattern with java. A few functions i will to perform as follows

  • Use breaker pattern to build acts proxy for service remote call.
  • Support config timeout and number of retry to execute command.
  • Compatible with multiple threads model.
  • Inheritance and flexibility.
  • Demo run with simple application.

UML for core implement Breaker pattern

circuit-breaker2

BreakerExecute.java is main class implement proxy acts for remote call

+ Return fail when state of service is OPEN

if (state == BreakerState.OPEN)
{
   try {
      error(new BreakerRejectException(), null);
      commandQueues.add(currentCommand);
      continue;
   }
      catch (Exception errHandle) {
      LOGGER.error(errHandle, errHandle);
   }
}

+ Change state to CLOSE when have a remote call is successful

currentCommand.run();
if (state == BreakerState.HALF_OPEN) {
   state = BreakerState.CLOSE;
   numberFailure.set(0);
}

+ Retry execute command with number of re-invoke is limited

Integer counter = mapRetryCommand.get(currentCommand.getId());
if(counter == null){
   counter = 1;
   mapRetryCommand.put(currentCommand.getId(), counter);
}
if (counter < numberAttempts) {
   commandQueues.add(currentCommand);
   counter++;
   mapRetryCommand.put(currentCommand.getId(), counter);
} else {
   mapRetryCommand.remove(currentCommand.getId());
try {
   error(new BreakerExpiredException(), currentCommand);
} catch (Exception errHandle) {
   LOGGER.error(errHandle, errHandle);
}

+ Change state CLOSE to OPEN when the number of recent failure exceed a specified threshold.

if (state == BreakerState.CLOSE) {
   if (numberFailure.incrementAndGet() > limitFailure) {
   state = BreakerState.OPEN;
   TIMER.newTimeout(new TimerTask() {
      @Override
      public void run(Timeout timeout) throws Exception {
      state = BreakerState.HALF_OPEN;
      }
   }, timeout, TimeUnit.MILLISECONDS);
}

Demo: Suppose, i have cluster with 2 server (A, B) and Server A will invoke addProduct() to B by remote call. Here i will implement circuit-breaker on A to manage invoke remote call to B.

circuit-breaker3

+ AddProductCommand is extended BreakerCommand

public class AddProductCommand extends BreakerCommand{
   private static Logger LOGGER =    Logger.getLogger(AddProductCommand.class);
   private Product product;

   @Override
   public void run() {
      LOGGER.info("Process product "+product.getName());
   }
   public Product getProduct() {
      return product;
   }
   public void setProduct(Product product) {
      this.product = product;
   }

+ ProductService is extended BreakerExecute and add AddProductCommand to execute

public class ProductService extends BreakerExecute{
   private static Logger LOGGER =    Logger.getLogger(ProductService.class);

   --more--

   public void addProduct(Product product){
      AddProductCommand command = new AddProductCommand();
      command.setProduct(product);
      putCommand(command);
   }
   --more--
}

In the next post i will implement circuit-pattern by Golang. Please contact to me or comment into this below about your opinion about post, thank you for watching.

Get more project : source code

Circuit Breaker pattern – Overview Part 1

In our system, we have many services and it is allocated at difference servers. We have used RPC (Remote Procedure Calls) to call between services but I have a problem about failure with remote call.In a distributed environment calls to remote resource and services can fail due to transient faults such as slow network connections, timeouts or the resources being over committed or temporary unavailable. So i started to learn about “retry pattern” and i have found “Circuit breaker” as pattern give me a solution for my problem.

I have set some goals as follows

+ Learn about “Circuit breaker pattern” model architecture and how to work.
+ Looking for some libraries support this pattern.
+ Practice pattern with Java and Golang.

In the part, i focus into overview about solution and explain how to it work ?

The Circuit pattern can prevent an application from repeatedly trying to execute an operation that’s likely to fail. Allowing it to continue without waiting for the fault to be fixed and it also enables an application to detect whether the fault has been resolved. The pattern will try execute operation and update status service when the problem to have been resolved.

Circuit breaker acts as a proxy for operations that might fail. The proxy should monitor the number of recent failure that have occurred, and use this information to decide whether to allow the operation to process.

A operation that might fail have 3 type state

Close :
+ The request invoke from application is routed to the operation.
+ Reset failure counter number of requests is failed.
+ If the number of recent failure exceed a specified threshold, state Close –> Open.

Open :
+ When operation change state from Close -> Open –> start timeout timer.
+ The request invoke from application fails immediately and an exception is returned to the application.
+ When this timeout timer is expired, state Open –> Half-Open.

Half-Open :
+ A limited number of requests from the application are allowed to pass through and invoke the operation.
+ If these requests are successful, state Half-Open –> Close.

circuit-breaker-diagram

When to use this pattern 
To prevent an application from trying to invoke a remote service or access a shared resource if this operation is highly likely to fail.

When should’t use this pattern 
Handling access to local private resource, such as in memory data structure.  As a substitute for handling exceptions in the business of your applications.

The next post, i will write how to implement circuit breaker pattern with java, thank you.