Friday, August 3, 2007

Achieving Proactive Business Activity Monitoring (BAM) solution by combining Dashboard with CEP

To some people think of BAM as a dashboard that display Key Business Performance Indicators (KBPIs). Mostly, these indicators are obtained from underlying technologies, databases, business applications and presented on the dashboard using nice looking graphs and gauges. While this typcal view is what some BAM vendors provide it falls far short of the actual business requirements:
  • Proactive correlation of complex interactions of key business systems
  • Detection of negative trends and prevention
  • Notifcation, proactive actions based on conditions and trends
  • Measuring impact of IT on key business services and the bottom line

So the dashboards provide the visualization part of the total BAM solution. The big question is how do you measure KBPIs reliably, and second how do you slice IT environment and its impact on those KBPIs. The later is a daunting task for most IT oragnizations -- trying to figure out how to obtain, measure and correlate thousands of different pieces of information to create a coherent picture of the impact on the business.

This where I beleive a solid CEP (Complex Event Processing) system might come to the rescue. Coupled with a solid data collection mechanism CEP can deliver what dashboards need in order to deliver a complete "Proactive BAM" solution.

So the case can be structured as follows: Use simple data collectors to obtain critical data required to make descisions (this could be tied to middleware, app servers, applications, business apps, existing tools), correlate information in CEP like engine to create KBPIs and integrate CEP/KBPIs with your favorite dashboard.

This process of course will only be successsful if both business and IT side are all on the same page -- meaning business requirements are well defined, and the process is well managed.

Thereore I beleive the key to any BAM implementation is combine dashboard technologies with CEP capabilities. The union of the two provides not just visualization, but cuasality, proactivity and drill down to the elements that impact your bottom line.

Java Runtime.exec() pitfalls re-examined

While there is plenty of stuff written about proper usage of Java Runtime.exec(), there still seems to be a problem with the way most developers use it. First the streams provided via Process object must be drained to prevent process hang and even deadlock, second upon process termination process streams must be closed (OutputStream, InputStream and ErrorStream). However even this clean up may not prevent your JVM from running out of file descriptors.

Apparently as tested with JRE 1.4.2_13 (Linux and AIX), JVM leaves open handles dangling upon process termination even if all streams are explicitly closed. Interestingly these handles are cleaned up when System.gc() is called explictly -- so it can not be a leak in the code.

As a result repeated exections of Runtime.exec() may cause descriptor exhaustion and subsequent failures when opening sockets, files, and launching programs from within your JVM. The common error is (too many files open).

After lots of tests, calling Process.destroy() after the process ends solves the handle leak.
However you must be very careful when calling this method, since you if you still have running threads that are reading/writting Process input streams , then destroy() would make them fails on the next IO. So thread synchronization must be put in place to make sure that destoy() is called only after all stream threads have terminated.

Not sure if this problem exists on all platforms and JVMs, but the lesson is that Runtime.exec is a lot more complicated then it seems and require careful handling within your code.