At the time, we chose storm because of a few reasons:

   - Metron inherited its codebase from OpenSOC, which chose Storm as it
   predated flink and spark streaming, the two other major contenders in the
   hadoop stack
   - Storm was battle tested at the time and, at least then, we had some
   concern around the maturity and performance of the other contenders
   - Specifically alternatives involving microbatching with high-throughput
   topologies like pcap was a concern

Going forward and looking back, I think it was a fine choice, honestly.
Ultimately, it quickly became apparent that it was necessary to produce a
simplified abstraction on top of the streaming technologies rather than
adopting their abstractions as the core point of extension for the
architecture.  So, for instance, we quickly realized that it was a better
experience for people to use Stellar to add enrichments rather than to
either modify code *or* flux files directly and possibly bounce
topologies.  This allowed us to not concern the user with the streaming
technology and also, fwiw, makes it easy to pivot to another streaming
technology if we so desire in the future.

Hope that provides some clarity!  Thanks for the very interesting question.


On Fri, Jan 12, 2018 at 6:29 PM, M. Aaron Bossert <[EMAIL PROTECTED]>