Mr Baseline’s EAI Crash Course – Part II

The Enterprise Service Bus (ESB)

These days, we refer to the hub as the ESB, or Enterprise Service Bus, and we clearly define its responsibilities:

1. Connectivity. The ESB should be able to transport data using one of a clearly defined small set of protocols, such as HTTP or some message queuing protocol (e.g. JMS).
2. Protocol Conversion. The ESB should be able to convert data sent over one protocol, so it can be forwarded over another.
3. Data Transformation. Different systems will send and receive data in different formats, such as fixed length flat files or XML. The ESB should be able to transform (or map) data freely between such formats.
4. Routing. A key feature of any sound integration architecture is loose coupling, meaning that the sending application should know as little as possible about the receiving application. Therefore, no data is sent directly between sender and receiver, but is always routed by the ESB.

These activities are collectively called Message Mediation. As simple as it may seem, designing and implementing an ESB around these four key capabilities is complex and requires careful design. Without a solid methodology, the full potential of the ESB will not be realized.

Methodology Key Features

After having completed a number of integration projects, you come to the not-so-startling conclusion that a lot of the stuff you do is done in pretty much the same way regardless of the systems involved. You map from one format to the other. You connect the ESB to various systems by reading and writing files in directories, or reading and writing records to database tables. You document your work. You deploy similar artifacts in similar ways to the same environments (e.g. test environment and production environment).

A sound methodology should capture those general components and make them available to all people involved in integration projects. Examples of such components include reusable software, document templates, procedures, and roles.

If you think you’ve seen this before, you may well be right. RUP (Rational Unified Process) is an example of such a methodology. RUP, however, is an all-purpose software engineering discipline, not immediately applicable to systems integration work (or any other specific software engineering endeavor, for that matter). What we will focus on in this series of posts are the specific features of a systems integration development methodology.

In the next post…

…I’ll introduce you to Baseline, the methodology we use at Zystems to deliver EAI solutions to our customers.

Mr Baseline’s EAI Crash Course – Part I

In the unlikely event that someone outside the circle of folks already familiar with the work we do at Zystems should find their way to this blog, I will spend the next couple of posts outlining our take on EAI.

Introduction

Integrating IT systems has traditionally been a complex and arduous task, and will remain so for the foreseeable future. Using sound methodology when tackling systems integration can substantially alleviate the complexity and avoid some of the pitfalls involved. This series of posts will outline some of the key features of such a methodology. It will also describes the Baseline methodology that enables an organization to get started with a fully implemented integration platform in 2-3 weeks.

The Problem

If you’ve ever tried to get more than a few IT systems to communicate with each other, you’re familiar with all the problems involved. Over time, you end up with an almost organic web of interdependencies. Obscure, poorly documented point-to-point connections (or interfaces), involving a bewildering collection of formats, protocols, and technologies, make up an infrastructure that only the most courageous dare tamper with.

One of the reasons for this is that most of the integration work has been done as part of the various projects dealing with developing and implementing the systems being integrated. This way, no coherent view on system integration exists. Each interface will be designed and implemented from scratch and choice of methods and technology will be entirely up to the people involved in the different projects.

As more and more demand is placed on the IT infrastructure to be flexible and responsive to new business demands (agile is a popular buzzword for this), the spaghetti approach becomes less and less satisfying.

Towards a Solution

Some years back, a way of dealing with this problem emerged in the shape of an architectural mode called Hub and Spoke. This architecture was built around the notion of having one central piece of software (the Hub) deal with all the complexities of inter-system communication. A new class of software, called Message Brokers, entered the stage, providing the technology to build those hubs.

Deploying this kind of middleware is, in itself, not a guaranteed way to integration nirvana. Although you now have a bunch of software components that can help you connect your systems with each other, nothing stops you from keeping the point-to-point mind-set, and you may end up creating a somewhat more coherent and smaller bowl of spaghetti. It will still lack the consistency that is required if you want control and transparency.

In the next post…

…I’ll introduce our view of the ESB and what you should expect from an EAI methodology.

A Sobering SOA Sermon

Through InfoQ I got another juicy lecture from QCon (I should go there next year!). Anne Thomas Manes puts SOA in its proper strategic perspective. It contains lots of gems for us disillusioned practitioners of SOA (in one of my presentations I have a slide with the heading “Why is it so damn hard?”). Well, when Anne puts the SOA life cycle time span in the 15-20 years bracket, maybe we shouldn’t weep if SOA doesn’t happen over night…It is good to keep in mind, as Anne points out, that the legacy we’re up against have been in the making for decades.

I bring two particular important bits of wisdom from this presentation. The first is the concept of applying a structured analysis of an enterprise’s application portfolio. Anne is talking a lot about the fat, unhealthy state of most IT organizations, basically originating in massive redundancy of functionality and data, driving obscene amounts of cost in integrating all those redundant bits and pieces. Anne suggests regularly applying a cost/benefit analysis to each application, and ruthlessly decommission any application not earning its keep, thereby shrinking the application portfolio. The other side of that coin, obviously, must be to identify any functionality the obsolete application provided and extract that functionality (as a reusable service) from somewhere else, with the ultimate goal of only having one provider of a particular business capability. This will obviously take some time…

The other bit I bring with me is the analogy between SOA and a new, healthy life style. Just as it’s fairly easy to go on a diet, loose a few ponds, and then gain them again, plus a few extra pounds, kicking off SOA in fits and starts is easier than digging in, gaining strategic sponsorship from management and preparing for the long haul. Yet to be of any real use, persistence will be essential to success. The paradox here, as can be gleaned from Anne’s lecture, is that you can’t mount a massive, all-encompassing SOA attack on your enterprise, because of legacy, politics and funding constraints.

So all in all, SOA is damn hard, yet the promises it holds if you persevere are of such magnitude that you really have no choice in the long run but to get on the SOA band-wagon.

The Next Big Thingy?

With much apparent hullabaloo, IBM recently acquired AptSoft, a small US software vendor focusing on CEP (Complex Event Processing). Googling on AptSoft will generate reams of hits mostly linking to various IT publications commenting on the acquisition; here is but one of them. It’s funny how comments about this refer to how CEP is superior to “traditional BPM tools”, as if these were more or less commodity among the companies of the world, when the fact of the matter is that these tools are only starting to be employed by vanguard enterprises, at least from our vantage point here in the Nordics. Maybe it’s different in other parts of the world, although I wouldn’t bet on it.

In typical IBM fashion, this is also SOA. And eventually, since we’re an IBM partner in the SOA sphere, this will be another technology to investigate. Hooray!

Plumbing for SOA

For quite some time, we’ve been promoting the image that what we’re mostly about at Zystems is plumbing. We even have a presentation called “Plumbing for SOA”. This would indicate that we draw the pipes but do not necessarily have much contact with genuine business value (such as sinks an faucets). I ran across this analogy in a most illuminating article by Bobby Woolf of IBM. I never really made the distinction between a Message Bus and an Enterprise Service Bus as such, but I think the definition Bob is making makes a whole lot of sense, and it is one that we’ve somehow implicitly managed to place firmly in the center of our concept model. One of the cornerstones of this model is the Service concept. The beauty of it is that we distinguish between public services and private services. Those customers who basically use the middleware (in our case mostly WebSphere MQ and Message Broker, but increasingly also WebSphere ESB) for traditional, technically oriented point-to-point integrations will do this using private services. Private services are exactly what Bob is referring to when he talks about the Message Bus. Private services are whatever technical goo we need to apply to pass the correct bits to the correct destination. What we manage to do is to offer our customers a seamless (from a technical point of view) migration path from a Message Bus to an ESB, by simply applying some sound design principles to the services, thereby creating public (SOA) services.

There is (unfortunately some might say) still a massive need for a well structured, well documented Message Bus. Our job is to make sure that it can mature together with the customer into an ESB that will truly be a technical linchpin of an SOA. So, yes, we’re plumbers, but when the customer is ready, magically there will appear sinks and faucets firmly (i.e. loosely ) attached to the pipes.

Getting paid

I occasionally suffer from a bad back. Many years ago, I friend recommended a chiropractor, who I have been going to over the years. This guy is amazing. Apart from the fact that he offers quick, high quality service, he manages to squeeze in an impressive number of clients each hour. To be specific, he’s scheduling system allows for 12 five minute slots each hour, at 500 SEK a pop. You just do the quick calculation in your head, and you realize this guy has a regular money press going for him.

It wasn’t always like this. When I started going to him, he answered the door himself, answered the phone, jotted down appointments, and he did it all out of a single room. The actual treatment was still quick and accurate, only he couldn’t squeeze as many clients into his schedule. So what changed? Process is what changed. He hired a nurse to take care of all his administrative tasks, got himself a computer system and got an extra room. The extra room is genius. It means that while I get dressed after the treatment, he’s already off treating the next guy in the next room. He probably doubled his capacity by streamlining his process. More to the point, he doubled his hourly intake of money, thereby allowing him to get richer, but also allowing him to pay for his process improvements.

OK, you ask, what’s this story doing in your blogg? Well, if this guy had been an IT consultant, I would have evaluated him by his hourly rate. He would have cost a whopping 6.000 SEK an hour. As a diligent purchaser, I would have compared him to other suppliers. As a matter of fact, in this case there was another guy I tried a couple of times. He would massage my back and then finish of with the same kind of back-breaking maneuvers that the 6.000 SEK guy does, only the whole procedure was closer to 45 minutes, and, sadly, it didn’t help a lot. It would probably cost me something like 400 SEK today, which quickly translates to 600 SEK an hour. Since I am a professional purchaser that take pride in my trade, I go for the 600 SEK/hr supplier. I don’t really care that I get a service that is inferior and takes longer time (45 minutes vs. 5 minutes).

The moral of the story (in case you missed it) is simply this: I, as the customer, would do well to compare the service I get (quick, efficient back-ache alleviation) , and the price I pay for it (500 SEK vs 400 SEK), and ask myself what is the value in this for me? Well, the answer, of course, is a no-brainer in this example. I go for the 6.000 SEK/hr guy any day.

So will integration services in any way resemble chiropractor ditto’s? Because that’s what we’re betting on !

Debunking the ESB

Anyone who’s been struggling with SOA would do well to spend some time together with Jim Webber when he talks about Guerilla SOA. For a company that promotes the ESB concept like Zystems does, the guy delivers some heavy punches. Nevertheless, I believe there are some really important lessons for us in his message. When we talk about the virtues of the ESB, we do emphasize the key benefit of accessing back-end legacy data. What I think we would all do well to consider is the ‘upper level’, i.e. where the business services reside. What Jim’s driving at, I think, is to make sure that anything on this level is standards based and open, not proprietary, which makes a whole lot of sense to me. And in a future where most applications have standards based interfaces, what, indeed, would be the purpose of a proprietary chunk of software in the middle? I do believe, however, that while we have lots of legacy apps to integrate, and lots of it outside of the SOA domain, methodologies to handle the ESB will be very much in demand.

But his most central insight, I think, is that all our efforts are “spaghetti agnostic”, i.e. our schemes and plans defy the spaghettiness of reality. We hope that magnificent tools and large-scale methodologies will somehow defeat the inherent messiness of reality. What we need is SOA – Spaghetti Oriented Architecture – hence his use of the term Guerilla SOA. I understand it as once and for all giving up the One Big Scheme pipe dream and find ways to embrace change. Quite a mouthful when you make your living trying to bring some orderliness to the integration world .

As an aside, it warms my heart to hear Jim talk about the ESB as an Erroneous Spaghetti Box! That’s exactly what we claim might happen if you don’t have a simple and straight-forward way of developing integrations (such as Baseline). Never mind that the guy wants to get rid of the ESB altogether, or nearly so. As for governance, he promotes the idea of a wiki based solution, not a high-tech UDDI-monster. Again, hello Baseline Service Registry (our wiki-based documentation).

Changing direction

Since we’re all excited at Zystems these days because we recently have embarked upon a journey toward a new business model, I’m reminded of the Sigmoid Growth Curve, a concept a former colleague of mine introduced me to back in th 90’s. I found a pretty good, condensed description of it. It’s implications, I think, are fundamental. Having just started our own efforts to break old habits and create new ones, it is a timely reminder of both the perils and promises of trying to change.

On the soft side of integration

So I got myself a blog. The plan is to blog on the professional side of my life. Which means I have to figure out where I stand in the world of EAI, SOA and the like, since that’s what we’re all about at Zystems. Thing is, I started out at Zystems doing strictly development work. Since I had done that for the last decades (not only at Zystems, though), I made a conscious decision a couple years back to drop the tech stuff more and more and go for the softer spots. Which means that I ended up managing our Systems Integration Concept, Baseline (hence the name of this blog). Colleagues and peers (check the blogroll) will no doubt keep pounding out tech stuff, so that will not be my niche. Instead, I’ll use this blog to help myself outline my new professional profile. What, if anything, is my contribution in the fuzzy area that deals with things like training, product development, conceptualization, project management, pre-sale, and all the other odd bits and pieces that I find myself doing at the job?