+
Jeff Bachtel 2013-01-15, 01:57
+
Connor Woodson 2013-01-15, 02:06
+
Jeff Bachtel 2013-01-15, 02:12
+
Connor Woodson 2013-01-15, 02:17
+
Brock Noland 2013-01-15, 02:23
+
Jeff Bachtel 2013-01-15, 17:12
-
Re: Unable to activate flume-ng-cassandra-sinkBrock Noland 2013-01-15, 17:21
Hi,
We haven't documented it yet as it's not in any release. I've been
meaning to do that! :)
I just created https://issues.apache.org/jira/browse/FLUME-1845 for this.
Brock
On Tue, Jan 15, 2013 at 9:12 AM, Jeff Bachtel <[EMAIL PROTECTED]> wrote:
> No, I hadn't separated the jarfiles out in that manner (I didn't realize the
> order they appeared in the CLASSPATH mattered). I'll certainly go change
> that now.
>
> Is there documentation on the plugins.d directory anywhere? I discovered it
> in the code by accident, and it met my needs while using this custom sink.
>
> Jeff
>
>
> On Mon, Jan 14, 2013 at 9:23 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>>
>> Great to hear the plugin directory is being used. Just to confirm, do
>> you have the following:
>>
>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink/lib/
>>
>> correct?
>>
>> Generally the plugin.d was created so this:
>>
>> /usr/local/flume/plugins.d/plugin/lib/
>>
>> would have the plugin jar and this:
>>
>> /usr/local/flume/plugins.d/plugin/libext/
>>
>> would have any jars the plugin depended on.
>>
>> Brock
>>
>> On Mon, Jan 14, 2013 at 6:17 PM, Connor Woodson <[EMAIL PROTECTED]>
>> wrote:
>> > The reason I don't recommend using it is that I didn't know it existed
>> > :P
>> >
>> > That seems like a great place to put it. I've done quite a bit of
>> > customization, so for me it's easier to just point Flume to the ./target
>> > directory of my java projects. But the plugins.d seems like a great
>> > place
>> > for third-party libraries.
>> >
>> > Glad I could help!
>> >
>> > - Connor
>> >
>> >
>> > On Mon, Jan 14, 2013 at 6:12 PM, Jeff Bachtel <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> Awesome, thanks; it was that typo.
>> >>
>> >> The plugins.d directory seems to be working, is there a particular
>> >> reason
>> >> you don't recommend using? I'm brand-spanking new to Flume, so I'm
>> >> curious.
>> >>
>> >> Thanks again,
>> >>
>> >> Jeff
>> >>
>> >>
>> >> On Mon, Jan 14, 2013 at 9:06 PM, Connor Woodson
>> >> <[EMAIL PROTECTED]>
>> >> wrote:
>> >>>
>> >>> Leave the Cassandra sink jar with its other jars in a folder some
>> >>> where.
>> >>> In FLUME_HOME/conf, open flume-env.sh and go to the FLUME_CLASSPATH
>> >>> line.
>> >>> Uncomment it if needed. Append the path to your cassandra sink and its
>> >>> associated jars, and see if that works.
>> >>>
>> >>> There doesn't appear to be a version incompatability; simply, the
>> >>> Cassandra jar is not found in the classpath. And the reason for that
>> >>> is it
>> >>> looks like you have a typo in your flume.conf.
>> >>>
>> >>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>
>> >>> should probably be
>> >>>
>> >>> com.btoddb.flume.sinks.cassandra.CassandraSink
>> >>>
>> >>> - Connor
>> >>>
>> >>>
>> >>> On Mon, Jan 14, 2013 at 5:57 PM, Jeff Bachtel <[EMAIL PROTECTED]>
>> >>> wrote:
>> >>>>
>> >>>> I have a build of Flume 1.4.0 on FreeBSD 8, installed into
>> >>>> /usr/local/flume . I manually created
>> >>>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink and untarred
>> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local
>> >>>> pull/build
>> >>>> of btoddb's sources on github), so I have a lib/ directory that's
>> >>>> completely
>> >>>> populated with both the sink plugin and any possible jarfiles it
>> >>>> might rely
>> >>>> on.
>> >>>>
>> >>>> I've likewise tried just copying
>> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar to flume/lib/ for it to
>> >>>> get
>> >>>> picked up.
>> >>>>
>> >>>> When I have a CassandraSink define in flume.conf as such:
>> >>>> webserver.sinks.cassandraSink.type >> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>> webserver.sinks.cassandraSink.channel = memoryChannel
>> >>>> webserver.sinks.cassandraSink.hosts = localhost
>> >>>> webserver.sinks.cassandraSink.cluster-name = MyCluster
>> >>>> webserver.sinks.cassandraSink.keyspace-name = ApacheLogs
>> >>>> webserver.sinks.cassandraSink.records-coldfam = Requests
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
We haven't documented it yet as it's not in any release. I've been
meaning to do that! :)
I just created https://issues.apache.org/jira/browse/FLUME-1845 for this.
Brock
On Tue, Jan 15, 2013 at 9:12 AM, Jeff Bachtel <[EMAIL PROTECTED]> wrote:
> No, I hadn't separated the jarfiles out in that manner (I didn't realize the
> order they appeared in the CLASSPATH mattered). I'll certainly go change
> that now.
>
> Is there documentation on the plugins.d directory anywhere? I discovered it
> in the code by accident, and it met my needs while using this custom sink.
>
> Jeff
>
>
> On Mon, Jan 14, 2013 at 9:23 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>>
>> Great to hear the plugin directory is being used. Just to confirm, do
>> you have the following:
>>
>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink/lib/
>>
>> correct?
>>
>> Generally the plugin.d was created so this:
>>
>> /usr/local/flume/plugins.d/plugin/lib/
>>
>> would have the plugin jar and this:
>>
>> /usr/local/flume/plugins.d/plugin/libext/
>>
>> would have any jars the plugin depended on.
>>
>> Brock
>>
>> On Mon, Jan 14, 2013 at 6:17 PM, Connor Woodson <[EMAIL PROTECTED]>
>> wrote:
>> > The reason I don't recommend using it is that I didn't know it existed
>> > :P
>> >
>> > That seems like a great place to put it. I've done quite a bit of
>> > customization, so for me it's easier to just point Flume to the ./target
>> > directory of my java projects. But the plugins.d seems like a great
>> > place
>> > for third-party libraries.
>> >
>> > Glad I could help!
>> >
>> > - Connor
>> >
>> >
>> > On Mon, Jan 14, 2013 at 6:12 PM, Jeff Bachtel <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> Awesome, thanks; it was that typo.
>> >>
>> >> The plugins.d directory seems to be working, is there a particular
>> >> reason
>> >> you don't recommend using? I'm brand-spanking new to Flume, so I'm
>> >> curious.
>> >>
>> >> Thanks again,
>> >>
>> >> Jeff
>> >>
>> >>
>> >> On Mon, Jan 14, 2013 at 9:06 PM, Connor Woodson
>> >> <[EMAIL PROTECTED]>
>> >> wrote:
>> >>>
>> >>> Leave the Cassandra sink jar with its other jars in a folder some
>> >>> where.
>> >>> In FLUME_HOME/conf, open flume-env.sh and go to the FLUME_CLASSPATH
>> >>> line.
>> >>> Uncomment it if needed. Append the path to your cassandra sink and its
>> >>> associated jars, and see if that works.
>> >>>
>> >>> There doesn't appear to be a version incompatability; simply, the
>> >>> Cassandra jar is not found in the classpath. And the reason for that
>> >>> is it
>> >>> looks like you have a typo in your flume.conf.
>> >>>
>> >>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>
>> >>> should probably be
>> >>>
>> >>> com.btoddb.flume.sinks.cassandra.CassandraSink
>> >>>
>> >>> - Connor
>> >>>
>> >>>
>> >>> On Mon, Jan 14, 2013 at 5:57 PM, Jeff Bachtel <[EMAIL PROTECTED]>
>> >>> wrote:
>> >>>>
>> >>>> I have a build of Flume 1.4.0 on FreeBSD 8, installed into
>> >>>> /usr/local/flume . I manually created
>> >>>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink and untarred
>> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local
>> >>>> pull/build
>> >>>> of btoddb's sources on github), so I have a lib/ directory that's
>> >>>> completely
>> >>>> populated with both the sink plugin and any possible jarfiles it
>> >>>> might rely
>> >>>> on.
>> >>>>
>> >>>> I've likewise tried just copying
>> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar to flume/lib/ for it to
>> >>>> get
>> >>>> picked up.
>> >>>>
>> >>>> When I have a CassandraSink define in flume.conf as such:
>> >>>> webserver.sinks.cassandraSink.type >> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>> webserver.sinks.cassandraSink.channel = memoryChannel
>> >>>> webserver.sinks.cassandraSink.hosts = localhost
>> >>>> webserver.sinks.cassandraSink.cluster-name = MyCluster
>> >>>> webserver.sinks.cassandraSink.keyspace-name = ApacheLogs
>> >>>> webserver.sinks.cassandraSink.records-coldfam = Requests
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
