On a recent project, I had the opportunity to work with the Mule ESB (Enterprise Service Bus) and started questioning the use of the ESB and more generally, when an ESB should be included in an ECM solution.
I found an interesting article online by MuleSoft founder Ross Mason (https://blogs.mulesoft.org/to-esb-or-not-to-esb/) that described some criteria to use when deciding if an ESB should be used. Using an ESB has a certain cachet associated with it in today’s enterprise architecture domain. Mason suggests that there are a number of enterprise architects that have advocated use of an ESB just to increase the number of acronyms on their CV.
With this perspective in mind, I looked at Mason’s list of criteria and applied it to the current project to see if an ESB was a necessary and appropriate part of the solution architecture. The project was suffering from scaling and complexity issues and initial investigations into these issues suggested that the Mule ESB was a major bottleneck in the system. Here’s the report card I developed for the system:
1) Are there more than three applications or services that need to be integrated?
- In this application, there were more than ten major services that were used by the front end but I questioned how many were really integrated at a deeper level than just a client calling a service. A more detailed review of the messaging architecture showed that there were a handful of instances when service calls were chained together to get the end results for the client.
- Did this one pass the test for using the ESB? My vote on this is yes, the ESB is useful to manage asynchronous chaining of messages between multiple back end, disparate services.
2) Is more than one communication protocol used to communicate to the services?
- Technically, yes. One was to send out email messages (SMTP) and all other communication was over HTTP or HTTPS and could have easily been integrated into the client. So did this pass the ESB test of having an abstraction layer to hide disparate messaging protocols? My vote here is a qualified no.
3) Is message routing or aggregation needed by the processing of the services?
- Yes. There were many instances of calling services and aggregating or splitting the results and then calling the same service to get more detailed information.
So my score card is roughly 66% in favor of using an ESB. The ESB was useful as a means of reducing the complexity of the communications surfaced to the client components. But it was interesting to see that after this somewhat quantitative assessment, the need for an ESB was still somewhat equivocal.
Having decided that the ESB was useful, we were still faced with the scaling and complexity issues we were seeing in the system. After doing a design review, we recognized that we had created a problem through a somewhat heads-down approach to incremental development. In the initial ESB deployment, only a few services were involved and the ESB application for managing the messaging was performing well. However, as the ESB code was built up over time, we hadn’t realized that the integration application was growing to the point of near instability. Following another recommendation from Ross’ article, we attempted as a side project to break up the monolithic application into multiple smaller applications. This required cross communications between the smaller ESB apps utilizing a JMS (Java Messaging Service) which is not integrated into the product due to the use of clustering of the servers. This was a huge undertaking and at the point in time of the project that this was attempted, there just wasn’t time or budget to complete. From our experience, the monolithic application consumes huge amounts memory, is harder to maintain and debug, and also causes the Mule Management Console (MMC) to become nearly impossible to use. So, the lesson learned was to get back to the agile basics and refactor along the way! After having seen the strengths and weaknesses of an ESB as part of the enterprise architecture on this project, we formalized two design criteria that we’re now using on our projects:
- Explicitly consider the need for an ESB and only introduce one if the ESB solves a significant set of messaging issues.
- Refactor the ESB implementation as the project grows in scope, recognizing that ESB performance and stability is a key consideration in overall system performance and stability.