Challenges faced, in design, deployment and management of in-motion analytics from high velocity streaming data, are unique and complex. These solutions should deal with ingesting data from variety of sources which may be incompatible or out of sync or produce data at different rate. The chances of failure of data sources or computing elements are very high and implementation should be able to compensate and recover from such failures while still maintaining high throughput and low latency.

We have successfully implemented many streaming analytics solutions for many of our clients using the IBM Streams product. Streams has many features and capabilities required for implementing challenging and advanced real-time analytics solutions. Some of Streams capabilities include application resiliency which ensures at least once processing of tuples, tag- based host deployment which ensures automated quick recovery, windowing capabilities for deduplication, merging and aggregation and use of annotations for parallelization. Streams also has out-of- the-box console for managing and monitoring deployments and also provides JMX extension for integrating with custom management frameworks.

In this presentation, we will go through some common design problems which we encountered in our projects and how they were solved using Streams. We will demonstrate what design and implementation choices you need to anticipate in such projects and common concepts and capabilities which are required in such enterprise solutions and platforms.