Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> HTTPSource with JSONHandler


Copy link to this message
-
HTTPSource with JSONHandler
Hello All,

I am trying to setup a HTTPSource with the built-in JSONHandler. Per the user guide :
http://flume.apache.org/FlumeUserGuide.html#jsonhandler
It is not clear if the data sent to flume has to contain the 'Header' and 'Body' part clearly called out? In that case, if I am getting my JSON from a different system then do I have to actually add the header and body strings and package it before sending to Flume. If so, then it could cause a lot of overhead to do that.
Example:
[
 {
  "headers" : {
             "timestamp" : "434324343",
             "host" : "random_host.example.com"
             },
  "body" : "random_body"
  },
  {
  "headers" : {
             "namenode" : "namenode.example.com",
             "datanode" : "random_datanode.example.com"
             },
  "body" : "really_random_body"
  }
]

So I have a huge JSON (XML actually but for now I am converting it to Json as I would need a XMLHandler written) payload that I need to send to Flume. Do I have to wrap the Json payload into the "body" tag?

In other words, given a standard Json, how do I convert it to Json format expected by Flume?

The user guide also says"One way to create an event in the format expected by this handler, is to use JSONEvent provided in the Flume SDK and use Google Gson to create the JSON string using the Gson#fromJson(Object, Type) method. "
So does tho mean I have handle this event somewhere?

Thanks in advance.