Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - flume-cassandra


Copy link to this message
-
Re: flume-cassandra
Sri Ramya 2013-01-31, 10:56
thanks for ur support,
 i will get back to u for farther help
On Thu, Jan 31, 2013 at 12:40 PM, Ralph Goers <[EMAIL PROTECTED]>wrote:

> Running Flume-ng on one host (or cluster of hosts) and Cassandra on its
> own Cluster of hosts is correct.  Just as all the other Flume Sinks, the
> Flume Cassandra Sink has to run in Rlume. The Flume Sink writes to
> Cassandra using one of the supported protocols. I believe we are using
> Hector but you could use Astyanax, Thrift or I suppose even CQL.  You could
> start with https://github.com/btoddb/flume-ng-cassandra-sink and tailor
> it to meet your needs.
>
> Ralph
>
>
> On Jan 30, 2013, at 8:35 PM, Sri Ramya wrote:
>
>
>  I am running flume-ng on one host and Cassandra  on another host. How i
> have to connect it. Which host i have to install flume-casandra sink? Do i
> need to run flume-ng on the host which is having Cassandra?
>
> Please reply me...
>
>
> On Wed, Jan 30, 2013 at 12:54 AM, Ralph Goers <[EMAIL PROTECTED]>wrote:
>
>> We ended up writing our own custom sink for Cassandra so that we could
>> customize the fields, write to Cassandra indexes and multiple column
>> families.  We primarily use Cassandra to store log records and we insure
>> they can't be duplicated by generating a guid in the Appender that writes
>> to Flume (via Avro).  The guid is then used as the row key.
>>
>> What do you mean by separation?
>>
>> We use Log4j 2's Flume Appender with the RFC5424Layout and
>> StructuredDataMessages (see EventLogger).  All the individual items in the
>> StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to
>> the FlumeEvent data map.  The Cassandra Sink then writes these to
>> individual columns in Cassandra.  The Cassandra Sink also writes these
>> items to the appropriate indexes so that we can perform whatever queries
>> that perform well.  The ipAddress is definitely one of the items we are
>> interested in and capture in our indexes.
>>
>> Ralph
>>
>>
>>
>> On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote:
>>
>> hi Ralph
>>
>> I am able to integrate flume with Cassandra. But the problem is ' if i
>> send a single event to Cassandra to store, it is storing multiple row in
>> Cassnadra. Thats why i thought of shifting to logsandra. But its very
>> difficult to to use that logsandra plug in. Any way by your suggestionsNow
>> i am staring work with Flume-ng.
>>
>> Make me clear in one thing, Is there any separation of storing data in
>> Cassandra based on flume-ng agent(IP address)???
>>
>> Thank you
>>
>>
>> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <[EMAIL PROTECTED]
>> > wrote:
>>
>>> This is absolutely NOT correct.   We are using Flume NG to log to
>>> Cassandra.   We fought with Flume OG for months trying to get it to work
>>> but it was never stable.  The architecture of Flume OG had serious
>>> problems, which is why Flume NG was created.
>>>
>>> Ralph
>>>
>>>
>>>
>>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>>>
>>> Hi ,
>>> Can you give details like your input format and all ...
>>> I think going with flume 0.9.4 for cassandra is better.
>>>
>>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <[EMAIL PROTECTED]>wrote:
>>>
>>>> What do you mean by "collector"?
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <[EMAIL PROTECTED]>wrote:
>>>>
>>>>>  Thank you very much. But I need a collector in my application,
>>>>> flume-ng  does not have any collector. Thats why i started with Cloudeera
>>>>> flum. Now i am getting a problem with integrating flume collector with
>>>>> cassandra.  If any body tried it before please help me.
>>>>> thank in advance.
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> Hi Sri,
>>>>>> Cloudera originally created Flume, then contributed it to the Apache
>>>>>> Software Foundation (ASF), and continues to invest heavily into it under
>>>>>> the auspices of the ASF. The current generation of Flume is called Flume
>>>>>> NG. I encourage you to use the latest "NG" generation of Flume (version