On June 13, 2018 at 10:30:26, Simon Elliston Ball (
[EMAIL PROTECTED]) wrote:
That’s where something like the Nifi solution would come in...
With the PutEnrichment processor and a ProcessHttpRequest processor, you do
have a web service for loading enrichments.
We could probably also create a rest service end point for it, which would
make some sense, but there is a nice multi-source, queuing, and lineage
element to the nifi solution.
>> I won't have access to, hence will need someone to be able to send us
>>>> Carolyn: just call the flat file loader from a script processor...
>>> I didn't believe that would work in my environment. I'm pretty sure the
>> script has dependencies on various Metron JARs, not least for the row id
>> hashing algorithm. I suppose this would require at least a partial
>> of Metron alongside NiFi, and would introduce additional work on the
>> cluster for any Metron upgrade. In some (enterprise) environments there
>> might be separation of ownership between NiFi and Metron.
>>> I also prefer not to have a Java app calling a bash script which calls
>> new java process, with logs or error output that might just get
>> up invisibly. Somewhere down the line this could hold up effective
>>>> Simon: I have actually written a stellar processor, which applies
>> stellar to all FlowFile attributes...
>>>> Simon: what didn't you like about the flatfile loader script?
>>> The flatfile loader script has worked fine for me when prepping
>> enrichment data in test systems, however it was a bit of a chore to get
>> JSON configuration files set up, especially for "wide" data sources that
>> may have 15-20 fields, e.g. Active Directory.
>>> More broadly speaking, I want to embrace the streaming data paradigm
>> tried to avoid batch jobs. With the DNS example, you might imagine a
>> where the enrichment data is streamed based on DHCP registrations, DNS
>> update events, etc. In principle this could reduce the window of time
>> we might enrich a data source with out-of-date data.
>>> -----Original Message-----
>>> From: Carolyn Duby [mailto:[EMAIL PROTECTED]]
>>> Sent: 12 June 2018 20:33
>>> To: [EMAIL PROTECTED]
>>> Subject: Re: Writing enrichment data directly from NiFi with
>>> I like the streaming enrichment solutions but it depends on how you are
>> getting the data in. If you get the data in a csv file just call the
>> file loader from a script processor. No special Nifi required.
>>> If the enrichments don’t arrive in bulk, the streaming solution is
>>> Carolyn Duby
>>> Solutions Engineer, Northeast
>>> [EMAIL PROTECTED]
>>> Join my team!