Skip to main content

Event Stream Processor Matrix

sematext sematext on

We published our first ever UI-focused post on Top JavaScript Dynamic Table Libraries the other day and got some valuable feedback – thanks!

We are back to talking about the backend again.  Our Search Analytics and Scalable Performance Monitoring services/products accept, process, and store huge amounts of data.  One thing both of these services do is process a stream of events in real-time (and batch, of course).  So what solutions are there that help one process data in real-time and perform some operations on a rolling window of data, such as the last 5 or 30 minutes of incoming event stream?  We know of several solutions that fit that bill, so we decided to put together a matrix with essential attributes of those tools in order to compare them and make our pick.  Below is the matrix we came up with.  If you are viewing this on our site, the table is likely going to be too wide, but it should look find in a proper feed reader.

If you like working on systems that handle large volumes of data, like our Search Analytics and Scalable Performance Monitoring services, we are hiring world-wide.

Matrix part 1:

LicenseLanguageScalingAdd or change rules on the flyOther infra neededRule types
EsperGPL2, commercialjavaScale upyesnoneDeclarative, query-based
Drools FusionASL 2.0javaScale upyesnoneDeclarative, mostly rule based, but support queries too
FlumeBaseASL 2.0javaHorizontal: natural sharding on top of FlumeyesFlumeDeclarative, query-based
StormEPL 1.0clojureHorizontalCan be implemented on top of ZookeeperZeroMQ, ZookeeperProvides only low level primitives(like grouping). Rule engine should be implemented manually on top.
S4ASL 2.0javaHorizontalCan be implemented on top of ZookeeperZookeeperProvides set of low level primitives. Somehow correlation support via joins. Documentation have a “windowing” section, but it empty.
ActiveinsightCPAL 1.0, commercialjavaHorizontalyesDeclarative, Query-like
KafkaAPL 2.0javaHorizontalZookeeperSet of low level primitives

Matrix part 2:

Docs / examplesMaturityCommunityURLNotes
Espervery goodmature,
Drools Fusiongood3 years,
Stormexistsused in productiongrowing very fasttech.backtype.comgood deployment features
S4averagealpha, butused in productionmedium (will grow under ASF)
Kafkagoodused in productionsmall (will grow under ASF)

So there you have it – we hope you find this useful.  If you have any comments or questions, tweet us (@sematext) or leave a comment here.  If you like working on systems that handle large volumes of data, like our Search Analytics and Scalable Performance Monitoring services, we are hiring world-wide.