-jiras-
On June 13, 2018 at 10:30:26, Simon Elliston Ball (
[EMAIL PROTECTED]) wrote:

That’s where something like the Nifi solution would come in...

With the PutEnrichment processor and a ProcessHttpRequest processor, you do
have a web service for loading enrichments.

We could probably also create a rest service end point for it, which would
make some sense, but there is a nice multi-source, queuing, and lineage
element to the nifi solution.

Simon
wrote:
the
that
>> I won't have access to, hence will need someone to be able to send us
the
>> data.
>>>
>>>> Carolyn: just call the flat file loader from a script processor...
>>>
>>> I didn't believe that would work in my environment. I'm pretty sure the
>> script has dependencies on various Metron JARs, not least for the row id
>> hashing algorithm. I suppose this would require at least a partial
install
>> of Metron alongside NiFi, and would introduce additional work on the
NiFi
>> cluster for any Metron upgrade. In some (enterprise) environments there
>> might be separation of ownership between NiFi and Metron.
>>>
>>> I also prefer not to have a Java app calling a bash script which calls
a
>> new java process, with logs or error output that might just get
swallowed
>> up invisibly. Somewhere down the line this could hold up effective
>> troubleshooting.
>>>
>>>> Simon: I have actually written a stellar processor, which applies
>> stellar to all FlowFile attributes...
>>>
>>> Gulp.
>>>
>>>> Simon: what didn't you like about the flatfile loader script?
>>>
>>> The flatfile loader script has worked fine for me when prepping
>> enrichment data in test systems, however it was a bit of a chore to get
the
>> JSON configuration files set up, especially for "wide" data sources that
>> may have 15-20 fields, e.g. Active Directory.
>>>
>>> More broadly speaking, I want to embrace the streaming data paradigm
and
>> tried to avoid batch jobs. With the DNS example, you might imagine a
future
>> where the enrichment data is streamed based on DHCP registrations, DNS
>> update events, etc. In principle this could reduce the window of time
where
>> we might enrich a data source with out-of-date data.
>>>
>>> Charlie
>>>
>>> -----Original Message-----
>>> From: Carolyn Duby [mailto:[EMAIL PROTECTED]]
>>> Sent: 12 June 2018 20:33
>>> To: [EMAIL PROTECTED]
>>> Subject: Re: Writing enrichment data directly from NiFi with
PutHBaseJSON
>>>
>>> I like the streaming enrichment solutions but it depends on how you are
>> getting the data in. If you get the data in a csv file just call the
flat
>> file loader from a script processor. No special Nifi required.
>>>
>>> If the enrichments don’t arrive in bulk, the streaming solution is
better.
>>>
>>> Thanks
>>> Carolyn Duby
>>> Solutions Engineer, Northeast
>>> [EMAIL PROTECTED]
>>> +1.508.965.0584
>>>
>>> Join my team!

for

Metron.