Mr Baseline’s EAI Crash Course – Part VI

The final installment of this EAI crash course isn’t about EAI but about SOA. So what’s the connection? To be sure, EAI and SOA are not the same thing at all. In fact, they address different domains. EAI is very much a technical discipline, mostly very IT-centric, delivering the technical solution to how disparate systems should talk to each other. SOA, on the other hand is (or should be) mostly about delivering business value through a complete overhaul of the way a business is managed, where the architecting of the business is the starting point for the implementation of this architected design using IT.

This description, as many of you know, rings pretty hollow in the mundane corridors that many of us inhabit. Mostly, business management still view SOA as primarily an IT issue, and IT struggles to “deliver SOA” despite the fact that it simply can’t be fully done without extensive management commitment.

However, what can be done while we’re waiting for the rest of humanity to see the light is to start putting an SOA-enabling infrastructure in place. And this is where EAI and SOA converges, in the shape of the ESB. It turns out that a well structured, well documented ESB where the Service concept is part of the basic design patterns is an ideal foundation for SOA. There is a journey of learning to be done, where the technical aspects of designing, implementing, deploying and maintaining services can be mastered. Using Baseline, the community involved in EAI can prepare themselves technically and organizationally for SOA, so that when the business folk eventually are ready to start requiring some real business centric services, the skills and delivery capabilities are in place, and requirement realization can be swift.

We believe strongly in this approach to SOA, so much so that we formalized this in the Baseline SOA Roadmap. Its purpose is to help companies start on their SOA journey. It won’t be SOA at first, but the transition to SOA will be that much easier when the basic skill-set is in place.

The roadmap itself is pretty simple:

1. Put an ESB in place
2. Make sure it is designed and documented using a structured, service-centric methodology (such as Baseline)
3. In the day-to-day EAI work, start identifying and implementing reusable services, applying appropriate design patterns
4. Eagerly await the rest of the business to jump aboard!

We actually got inspired to do this by one of our clients who said: “We started of doing systems integration using Baseline, and when the SOA buzz arrived, we realized that, hey, we already have an SOA!” I would have to assume that they meant the SOA-enabling infrastructure – the organizational overhaul issues are squarly outside Baseline’s domain!

This post concludes the EAI crash course. If yo want to know more about Baseline, please visit our website.

Does SOA require an ESB?

For those of you not subscribing to InfoQ, I just want to point you to this article, dealing with one of the cornerstones of how we recommend our customers to design their ESB, with or without an SOA perspective: Don’t Let Consumers and Service Providers Communicate Directly. It ends with a conclusion that I see pop up over and over again (and one that I agree with): No, an ESB is not strictly required to build an SOA, but it can sure help.

Mr Baseline’s EAI Crash Course – Part III

Baseline

Baseline (developed and marketed by Zystems) is a methodology that distills tens of thousands of hours spent integrating IT systems of some of the largest enterprises in Scandinavia into a lean packaged Best Practice for implementing an ESB.

Concept Model

A sound ESB will have a coherent design that permeates all solutions deployed on it. In order to achieve this coherency, Baseline defines a small number of key objects and their relationships to each other. Every solution will be modeled using those objects. All document templates are structured using the same objects.
Integration work generates a myriad of artifacts, such as documents, source code files, script files, binaries, and so on. Naming these artifacts in a consistent way is no small feat (as anyone involved in designing naming conventions will testify). Structured around the concept model, Baseline includes a comprehensive set of Naming Conventions, ranging from the naming of documents, all the way through the naming of source code files, and how to structure and name folders in a versioning system.

An example of such an object is the Contract. The Contract explicitly defines the relationship between the Consumer (another object in the Concept Model) of a Service and the Service itself. In a future post I will present the concept model in more detail.

Documents and Templates

Documenting an ESB in a consistent way is crucial for achieving a number of important goals:

• Maintainability. Supervising and managing the ESB is a complex art, since it will involve a number of operating systems, servers, software components, and contacts with system owners. Having a set of consistently designed documents, all kept in the same place, describing all solutions deployed on the ESB is absolutely essential for this task.
• Reusability. As elaborated later in this white paper, a sound ESB is an ideal foundation (actually, it’s more or less a prerequisite) for building a SOA (Service Oriented Architecture). One of the key promises of SOA is the concept of reusable services. However, a service poorly documented, and hidden away on some shared disk in a network, is not likely to be identified by new integration projects on the lookout for reusable services. Having all services documented in a consistent manner in a central registry is a minimum requirement for service reuse to happen.
• Quality. Working with design and implementation supported by document templates ensures that all relevant questions are asked at an early stage (provided the creation of the documents is a living part of the design and implementation process rather than an afterthought).

Baseline provides a comprehensive set of templates and documents covering all aspects of an integration project, including:

• Requirements collection
• Project startup
• Solution design and implementation
• Testing
• Packaging and deploying solutions
• Supervising solutions in a production environment
• Governance (i.e. the centralized organizational function of maintaining consistency across integration projects)

In the next post…

…I’ll continue to explore the Baseline methodology.

Mr Baseline’s EAI Crash Course – Part II

The Enterprise Service Bus (ESB)

These days, we refer to the hub as the ESB, or Enterprise Service Bus, and we clearly define its responsibilities:

1. Connectivity. The ESB should be able to transport data using one of a clearly defined small set of protocols, such as HTTP or some message queuing protocol (e.g. JMS).
2. Protocol Conversion. The ESB should be able to convert data sent over one protocol, so it can be forwarded over another.
3. Data Transformation. Different systems will send and receive data in different formats, such as fixed length flat files or XML. The ESB should be able to transform (or map) data freely between such formats.
4. Routing. A key feature of any sound integration architecture is loose coupling, meaning that the sending application should know as little as possible about the receiving application. Therefore, no data is sent directly between sender and receiver, but is always routed by the ESB.

These activities are collectively called Message Mediation. As simple as it may seem, designing and implementing an ESB around these four key capabilities is complex and requires careful design. Without a solid methodology, the full potential of the ESB will not be realized.

Methodology Key Features

After having completed a number of integration projects, you come to the not-so-startling conclusion that a lot of the stuff you do is done in pretty much the same way regardless of the systems involved. You map from one format to the other. You connect the ESB to various systems by reading and writing files in directories, or reading and writing records to database tables. You document your work. You deploy similar artifacts in similar ways to the same environments (e.g. test environment and production environment).

A sound methodology should capture those general components and make them available to all people involved in integration projects. Examples of such components include reusable software, document templates, procedures, and roles.

If you think you’ve seen this before, you may well be right. RUP (Rational Unified Process) is an example of such a methodology. RUP, however, is an all-purpose software engineering discipline, not immediately applicable to systems integration work (or any other specific software engineering endeavor, for that matter). What we will focus on in this series of posts are the specific features of a systems integration development methodology.

In the next post…

…I’ll introduce you to Baseline, the methodology we use at Zystems to deliver EAI solutions to our customers.

Plumbing for SOA

For quite some time, we’ve been promoting the image that what we’re mostly about at Zystems is plumbing. We even have a presentation called “Plumbing for SOA”. This would indicate that we draw the pipes but do not necessarily have much contact with genuine business value (such as sinks an faucets). I ran across this analogy in a most illuminating article by Bobby Woolf of IBM. I never really made the distinction between a Message Bus and an Enterprise Service Bus as such, but I think the definition Bob is making makes a whole lot of sense, and it is one that we’ve somehow implicitly managed to place firmly in the center of our concept model. One of the cornerstones of this model is the Service concept. The beauty of it is that we distinguish between public services and private services. Those customers who basically use the middleware (in our case mostly WebSphere MQ and Message Broker, but increasingly also WebSphere ESB) for traditional, technically oriented point-to-point integrations will do this using private services. Private services are exactly what Bob is referring to when he talks about the Message Bus. Private services are whatever technical goo we need to apply to pass the correct bits to the correct destination. What we manage to do is to offer our customers a seamless (from a technical point of view) migration path from a Message Bus to an ESB, by simply applying some sound design principles to the services, thereby creating public (SOA) services.

There is (unfortunately some might say) still a massive need for a well structured, well documented Message Bus. Our job is to make sure that it can mature together with the customer into an ESB that will truly be a technical linchpin of an SOA. So, yes, we’re plumbers, but when the customer is ready, magically there will appear sinks and faucets firmly (i.e. loosely ) attached to the pipes.

Debunking the ESB

Anyone who’s been struggling with SOA would do well to spend some time together with Jim Webber when he talks about Guerilla SOA. For a company that promotes the ESB concept like Zystems does, the guy delivers some heavy punches. Nevertheless, I believe there are some really important lessons for us in his message. When we talk about the virtues of the ESB, we do emphasize the key benefit of accessing back-end legacy data. What I think we would all do well to consider is the ‘upper level’, i.e. where the business services reside. What Jim’s driving at, I think, is to make sure that anything on this level is standards based and open, not proprietary, which makes a whole lot of sense to me. And in a future where most applications have standards based interfaces, what, indeed, would be the purpose of a proprietary chunk of software in the middle? I do believe, however, that while we have lots of legacy apps to integrate, and lots of it outside of the SOA domain, methodologies to handle the ESB will be very much in demand.

But his most central insight, I think, is that all our efforts are “spaghetti agnostic”, i.e. our schemes and plans defy the spaghettiness of reality. We hope that magnificent tools and large-scale methodologies will somehow defeat the inherent messiness of reality. What we need is SOA – Spaghetti Oriented Architecture – hence his use of the term Guerilla SOA. I understand it as once and for all giving up the One Big Scheme pipe dream and find ways to embrace change. Quite a mouthful when you make your living trying to bring some orderliness to the integration world .

As an aside, it warms my heart to hear Jim talk about the ESB as an Erroneous Spaghetti Box! That’s exactly what we claim might happen if you don’t have a simple and straight-forward way of developing integrations (such as Baseline). Never mind that the guy wants to get rid of the ESB altogether, or nearly so. As for governance, he promotes the idea of a wiki based solution, not a high-tech UDDI-monster. Again, hello Baseline Service Registry (our wiki-based documentation).