Thanks for the responses. I appreciate the willingness to look at creating a NiFi processer. That would be great!
Just to follow up on this (after a week looking after the "ops" side of dev-ops): I really don't want to have to use the flatfile loader script, and I'm not going to be able to write a Metron-style HBase key generator any time soon, but I have had some success with a different approach.
1. Generate data in CSV format, e.g. "server.domain.local","A","192.168.0.198"
2. Send this to a HTTP listener in NiFi
3. Write to a kafka topic
I then followed your instructions in this blog:https://cwiki.apache.org/confluence/display/METRON/2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment
4. Create a new "dns" sensor in Metron
5. Use the CSVParser and SimpleHbaseEnrichmentWriter, and parserConfig settings to push this into HBase:
"shew.table": " dns",
And... it seems to be working. At least, I have data in HBase which looks more like the output of the flatfile loader.
From: Casey Stella [mailto:[EMAIL PROTECTED]]
Sent: 05 June 2018 14:56
To: [EMAIL PROTECTED]
Subject: Re: Writing enrichment data directly from NiFi with PutHBaseJSON
The problem, as you correctly diagnosed, is the key in HBase. We construct the key very specifically in Metron, so it's unlikely to work out of the box with the NiFi processor unfortunately. The key that we use is formed here in the codebase:https://github.com/cestella/incubator-metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/converter/EnrichmentKey.java#L51
To put that in english, consider the following:
- type - The enrichment type
- indicator - the indicator to use
- hash(*) - A murmur 3 128bit hash function
the key is hash(indicator) + type + indicator
This hash prefixing is a standard practice in hbase key design that allows the keys to be uniformly distributed among the regions and prevents hotspotting. Depending on how the PutHBaseJSON processor works, if you can construct the key and pass it in, then you might be able to either construct the key in NiFi or write a processor to construct the key.
Ultimately though, what Carolyn said is true..the easiest approach is probably using the flatfile loader.
If you do get this working in NiFi, however, do please let us know and/or consider contributing it back to the project as a PR :)
On Fri, Jun 1, 2018 at 6:26 AM Charles Joynt <[EMAIL PROTECTED]>