Event Monitoring, ESP/CEP and Emergent Architecture

ESP, or CEP, has come into the norm in recent years mainly – as most technologies of any value in the enterprise usually are – from academia. There's still plenty of confusion surrounding the topic. This entry is to discuss the circumstances surrounding a good CEP implementation and to ground the reader in the “why” of the discipline, not the “how” (not yet, anyway). The idea that a running system should be observed to gain insight about its application may seem obvious, but it's startlingly difficult to get right.

We are all used to reactive monitoring, where you take measures to guarantee that a system doesn't dip below certain service availability thresholds. For instance, Nagios, a popular open source tool designed for system monitoring, lets the user define scripts to “events” and then react to them. In these systems, events are usually things like an HTTP resource not responding to a ping, or a file mount not responding to a touch command. There is pre-packaged functionality, but the real power lies in being able to detect, and react to, custom events in the system to ensure uptime. This is useful in any number of situations. I personally have had to deal with it when integrating the IVR solution Asterix, as well as when integrating a third party's customer CRM solution which was prone to crashes.

Sometimes, we do reporting to verify the performance of a system. This type of monitoring is also geared as much towards opening a porthole into the system for business analysts as it is any technical objective. Often, this kind of functionality is geared towards providing data on a window of “expected” activity. You don't report activity in a system to asses glaring deviance from expected behavior, you report to account for activity: sales figures, charting, user sign up, growth, etc.

So, clearly, the need to introspect a running system exists, but the most sophisticated reach of this technology lies in using to distill growth and possibility. Put another way: why react when you can predict and act.

The market is constantly changing, and the cutting edge enterprise will want to stay ahead of its competitors. This requires real time event analysis. Doing so means you can react to market forces quicker, that you can adapt process as well as technology to the future. Thus, CEP can be an enabling component of emergent architectures. Indeed, Gartner has outlined 7 properties of emergent architectures, one of which is that they are non-deterministic, which is why it's important to trap the entropy in a system and analyze as effectively as possible. In the world of technology solutions that let you mine event streams in your enterprise for so called “complex events” - events synthesized by aggregating many smaller events – are wildly variant in their approach. Often, the tool will feature a rule engine that lets you position it to subscribe to many events and perform logic. Correlation between the events is, obviously, entirely enterprise-specific, and it is this correlation that CEP tools help you define.