Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Sinking data to a Node.js server/listener


+
Erik Bertrand 2013-02-17, 04:50
+
Bertrand Dechoux 2013-02-17, 22:43
+
Erik Bertrand 2013-02-21, 13:41
Copy link to this message
-
Re: Sinking data to a Node.js server/listener
Bertrand Dechoux 2013-02-21, 14:08
In the end a syslog event/message on tcp is only a tcp message, so indeed
it should be simple with a tcp server (like nodejs). BUT I did answer too
quickly. I have more experience with flume (og) than flume-ng. And it does
indeed look like there is no syslog sink for the new generation. So I am
afraid that writing a custom sink is the way to go...

Bertrand
On Thu, Feb 21, 2013 at 2:41 PM, Erik Bertrand <[EMAIL PROTECTED]> wrote:

> Hello Bertrand.  Thanks very much for the reply!  (I love that you we
> share a common first name/last name!)
>
> I was not aware that there is a syslog sink.  I see there's an HDFS sink
> and a "File Roll" sink, but those do not seem to utilize syslog (I'm very
> new to this, so I may be misguided).  How would I go about setting up such
> a sync - are you saying I would be able to post the data to an IP/port that
> Node.js is listening on?
>
> Again, many thanks for the assistance!
>
> Erik
>
>
>
> On Sun, Feb 17, 2013 at 5:43 PM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
>
>> Should it be assumed that structured data are transferred using flume and
>> that's why Avro is mentioned? One easy solution I can see would be to use a
>> syslog output that would actually point to your nodejs instance (and not a
>> real syslog endpoint).
>>
>> Regards
>>
>> Bertrand
>>
>>
>> On Sun, Feb 17, 2013 at 5:50 AM, Erik Bertrand <[EMAIL PROTECTED]> wrote:
>>
>>> I'm looking to get Flume data transferred to a Node.js server listening
>>> on a specific port.  I'm not interested in actually storing the data
>>> anywhere, as it's simply using that data to display "current activity" and
>>> doesn't need to be persist it anywhere.  I'm transferring a very small
>>> amount of data - just a series of IP addresses, actually.  So I'd like to
>>> keep the architecture simple, too.
>>>
>>> At first I was thinking I could use the Avro sink to send the data
>>> directly to Node.js using a dnode <https://npmjs.org/package/dnode> server
>>> object (i.e. RPC), but there seems to be more to it than the basic setup.
>>>  I'm just not sure how to configure the Node.js side to understand the Avro
>>> sink RPC request (or if that's even possible).
>>>
>>> I've been looking at creating a custom sink to do this; I've not written
>>> one before, much less written anything in Java, so that'd be new to me.
>>>  Any pointers?
>>>
>>> Erik
>>>
>>>
>>
>>
>> --
>> Bertrand Dechoux
>
>
>