Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> flume-cassandra


+
Sri Ramya 2013-01-23, 17:59
+
rektide 2013-01-23, 18:23
+
Camp, Roy 2013-01-23, 18:52
+
shekhar sharma 2013-01-24, 01:53
+
Sri Ramya 2013-01-24, 03:30
+
Mike Percy 2013-01-24, 04:56
+
Sri Ramya 2013-01-24, 05:05
+
Mike Percy 2013-01-24, 05:13
+
Sri Ramya 2013-01-24, 05:45
+
Mike Percy 2013-01-24, 08:25
+
Priyanka Jain 2013-01-24, 05:46
+
Sri Ramya 2013-01-24, 05:50
+
Ralph Goers 2013-01-28, 16:56
+
Sri Ramya 2013-01-29, 03:54
+
Sri Ramya 2013-01-29, 05:15
+
Alexander Alten-Lorenz 2013-01-29, 07:31
+
Ralph Goers 2013-01-29, 19:24
+
Yogi Nerella 2013-01-29, 19:43
+
Ralph Goers 2013-01-29, 21:59
+
Yogi Nerella 2013-01-29, 22:07
+
Sri Ramya 2013-01-31, 04:35
+
Ralph Goers 2013-01-31, 07:10
+
Sri Ramya 2013-01-31, 10:56
Copy link to this message
-
Re: flume-cassandra
hi

I am unable to talk with cassandra using flume-ng sink. Can you tell me
what is the configuration file to give to flume agent.
On Thu, Jan 31, 2013 at 4:26 PM, Sri Ramya <[EMAIL PROTECTED]> wrote:

> thanks for ur support,
>  i will get back to u for farther help
>
>
> On Thu, Jan 31, 2013 at 12:40 PM, Ralph Goers <[EMAIL PROTECTED]>wrote:
>
>> Running Flume-ng on one host (or cluster of hosts) and Cassandra on its
>> own Cluster of hosts is correct.  Just as all the other Flume Sinks, the
>> Flume Cassandra Sink has to run in Rlume. The Flume Sink writes to
>> Cassandra using one of the supported protocols. I believe we are using
>> Hector but you could use Astyanax, Thrift or I suppose even CQL.  You could
>> start with https://github.com/btoddb/flume-ng-cassandra-sink and tailor
>> it to meet your needs.
>>
>> Ralph
>>
>>
>> On Jan 30, 2013, at 8:35 PM, Sri Ramya wrote:
>>
>>
>>  I am running flume-ng on one host and Cassandra  on another host. How i
>> have to connect it. Which host i have to install flume-casandra sink? Do i
>> need to run flume-ng on the host which is having Cassandra?
>>
>> Please reply me...
>>
>>
>> On Wed, Jan 30, 2013 at 12:54 AM, Ralph Goers <[EMAIL PROTECTED]
>> > wrote:
>>
>>> We ended up writing our own custom sink for Cassandra so that we could
>>> customize the fields, write to Cassandra indexes and multiple column
>>> families.  We primarily use Cassandra to store log records and we insure
>>> they can't be duplicated by generating a guid in the Appender that writes
>>> to Flume (via Avro).  The guid is then used as the row key.
>>>
>>> What do you mean by separation?
>>>
>>> We use Log4j 2's Flume Appender with the RFC5424Layout and
>>> StructuredDataMessages (see EventLogger).  All the individual items in the
>>> StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to
>>> the FlumeEvent data map.  The Cassandra Sink then writes these to
>>> individual columns in Cassandra.  The Cassandra Sink also writes these
>>> items to the appropriate indexes so that we can perform whatever queries
>>> that perform well.  The ipAddress is definitely one of the items we are
>>> interested in and capture in our indexes.
>>>
>>> Ralph
>>>
>>>
>>>
>>> On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote:
>>>
>>> hi Ralph
>>>
>>> I am able to integrate flume with Cassandra. But the problem is ' if i
>>> send a single event to Cassandra to store, it is storing multiple row in
>>> Cassnadra. Thats why i thought of shifting to logsandra. But its very
>>> difficult to to use that logsandra plug in. Any way by your suggestionsNow
>>> i am staring work with Flume-ng.
>>>
>>> Make me clear in one thing, Is there any separation of storing data in
>>> Cassandra based on flume-ng agent(IP address)???
>>>
>>> Thank you
>>>
>>>
>>> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> This is absolutely NOT correct.   We are using Flume NG to log to
>>>> Cassandra.   We fought with Flume OG for months trying to get it to work
>>>> but it was never stable.  The architecture of Flume OG had serious
>>>> problems, which is why Flume NG was created.
>>>>
>>>> Ralph
>>>>
>>>>
>>>>
>>>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>>>>
>>>> Hi ,
>>>> Can you give details like your input format and all ...
>>>> I think going with flume 0.9.4 for cassandra is better.
>>>>
>>>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> What do you mean by "collector"?
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>>  Thank you very much. But I need a collector in my application,
>>>>>> flume-ng  does not have any collector. Thats why i started with Cloudeera
>>>>>> flum. Now i am getting a problem with integrating flume collector with
>>>>>> cassandra.  If any body tried it before please help me.
>>>>>> thank in advance.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <[EMAIL PROTECTED]>wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB