Flume, mail # user - Re: Could not instantiate class org.apache.flume.clients.log4jappender

Could not instantiate class org.apache.flume.clients.log4jappender

孙东亮 2013-01-14, 03:25

I'm a newbie for the Flume, and I just set up flume for a test.
I want to use the log4jappender to get log info from PIG scripts, the
log4j.properties is:

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = localhost
log4j.appender.flume.Port = 41414
log4j.logger.org.apache.pig=DEBUG,flume

And for the flume.config:

a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 41414

# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

But I got the following error when run a pig script:
java.lang.ClassNotFoundException:
org.apache.flume.clients.log4jappender.Log4jAppender
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
at
org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
at
org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
at
org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
at
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
at
org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
at
org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
at org.apache.pig.Main.configureLog4J(Main.java:678)
at org.apache.pig.Main.run(Main.java:337)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
log4j:ERROR Could not instantiate appender named "flume".

Could you please help me find what I miss or where is incorrect for the
configuration.

Thanks a lot!
Dongliang

RE: Could not instantiate class org.apache.flume.clients.log4jappender

Chhaya Vishwakarma 2013-01-14, 04:52

Put flume-log4j-appender<https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-log4j-appender/0.9.4-cdh3u1/flume-log4j-appender-0.9.4-cdh3u1.jar>.jar in CLASSPATH. Jar can be found in FLUME_HOME/lib/

From: 孙东亮 [mailto:[EMAIL PROTECTED]]
Sent: Monday, January 14, 2013 8:55 AM
To: [EMAIL PROTECTED]
Subject: Could not instantiate class org.apache.flume.clients.log4jappender
I'm a newbie for the Flume, and I just set up flume for a test.
I want to use the log4jappender to get log info from PIG scripts, the log4j.properties is:

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender

log4j.appender.flume.Hostname = localhost

log4j.appender.flume.Port = 41414

log4j.logger.org.apache.pig=DEBUG,flume

And for the flume.config:

a1.sources = r1

a1.sinks = k1

a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = avro

a1.sources.r1.bind = localhost

a1.sources.r1.port = 41414

# Describe the sink

a1.sinks.k1.type = logger

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

But I got the following error when run a pig script:
java.lang.ClassNotFoundException: org.apache.flume.clients.log4jappender.Log4jAppender
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
at org.apache.pig.Main.configureLog4J(Main.java:678)
at org.apache.pig.Main.run(Main.java:337)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
log4j:ERROR Could not instantiate appender named "flume".

Could you please help me find what I miss or where is incorrect for the configuration.

Thanks a lot!
Dongliang

________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Dongliang Sun 2013-01-14, 05:24

Actually I have append the FLUME_HOME/lib/ to my CLASSPATH.
Whether there is some problem in my PIG log4j.properties?
Do you have any experience using log4jappender on PIG?

Thanks,
Dongliang
2013/1/14 Chhaya Vishwakarma <[EMAIL PROTECTED]>

> Put flume-log4j-appender<https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-log4j-appender/0.9.4-cdh3u1/flume-log4j-appender-0.9.4-cdh3u1.jar>.jar
> in CLASSPATH. Jar can be found in FLUME_HOME/lib/****
>
> ** **
>
> *From:* 孙东亮 [mailto:[EMAIL PROTECTED]]
> *Sent:* Monday, January 14, 2013 8:55 AM
> *To:* [EMAIL PROTECTED]
> *Subject:* Could not instantiate class
> org.apache.flume.clients.log4jappender****
>
> ** **
>
> I'm a newbie for the Flume, and I just set up flume for a test.
> I want to use the log4jappender to get log info from PIG scripts, the
> log4j.properties is:****
>
> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender ****
>
> log4j.appender.flume.Hostname = localhost ****
>
> log4j.appender.flume.Port = 41414 ****
>
> log4j.logger.org.apache.pig=DEBUG,flume****
>
> And for the flume.config: ****
>
> a1.sources = r1 ****
>
> a1.sinks = k1 ****
>
> a1.channels = c1 ****
>
> ** **
>
> # Describe/configure the source****
>
> a1.sources.r1.type = avro ****
>
> a1.sources.r1.bind = localhost ****
>
> a1.sources.r1.port = 41414 ****
>
> ** **
>
> # Describe the sink ****
>
> a1.sinks.k1.type = logger ****
>
> # Use a channel which buffers events in memory ****
>
> a1.channels.c1.type = memory ****
>
> a1.channels.c1.capacity = 1000 ****
>
> a1.channels.c1.transactionCapacity = 100 ****
>
> ** **
>
> # Bind the source and sink to the channel ****
>
> a1.sources.r1.channels = c1 ****
>
> a1.sinks.k1.channel = c1 ****
>
> But I got the following error when run a pig script:
> java.lang.ClassNotFoundException:
> org.apache.flume.clients.log4jappender.Log4jAppender
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:169)
> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
> at
> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
> at
> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
> at
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
> at
> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
> at org.apache.pig.Main.configureLog4J(Main.java:678)
> at org.apache.pig.Main.run(Main.java:337)
> at org.apache.pig.Main.main(Main.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> log4j:ERROR Could not instantiate appender named "flume".****
>
> Could you please help me find what I miss or where is incorrect for the
> configuration.****
>
> Thanks a lot!
> Dongliang****
>
> ------------------------------
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Connor Woodson 2013-01-14, 05:50

You could try calling 'REGISTER <log4jappender jar>' at the start of your
script, or on the command line by adding
-Dpig.additional.libraries=<log4jappender jar> to your Pig call. I don't
know if that will work or not though.

You could try placing it in pig's lib folder.

You also need to include the flume-ng-sdk jar the same way as the
log4jappender one, as the sdk is a dependency of the appender.
On Sun, Jan 13, 2013 at 9:24 PM, Dongliang Sun <[EMAIL PROTECTED]> wrote:

> Actually I have append the FLUME_HOME/lib/ to my CLASSPATH.
> Whether there is some problem in my PIG log4j.properties?
> Do you have any experience using log4jappender on PIG?
>
> Thanks,
> Dongliang
>
>
>
> 2013/1/14 Chhaya Vishwakarma <[EMAIL PROTECTED]>
>
>> Put flume-log4j-appender<https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-log4j-appender/0.9.4-cdh3u1/flume-log4j-appender-0.9.4-cdh3u1.jar>.jar
>> in CLASSPATH. Jar can be found in FLUME_HOME/lib/****
>>
>> ** **
>>
>> *From:* 孙东亮 [mailto:[EMAIL PROTECTED]]
>> *Sent:* Monday, January 14, 2013 8:55 AM
>> *To:* [EMAIL PROTECTED]
>> *Subject:* Could not instantiate class
>> org.apache.flume.clients.log4jappender****
>>
>> ** **
>>
>> I'm a newbie for the Flume, and I just set up flume for a test.
>> I want to use the log4jappender to get log info from PIG scripts, the
>> log4j.properties is:****
>>
>> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender ****
>>
>> log4j.appender.flume.Hostname = localhost ****
>>
>> log4j.appender.flume.Port = 41414 ****
>>
>> log4j.logger.org.apache.pig=DEBUG,flume****
>>
>> And for the flume.config: ****
>>
>> a1.sources = r1 ****
>>
>> a1.sinks = k1 ****
>>
>> a1.channels = c1 ****
>>
>> ** **
>>
>> # Describe/configure the source****
>>
>> a1.sources.r1.type = avro ****
>>
>> a1.sources.r1.bind = localhost ****
>>
>> a1.sources.r1.port = 41414 ****
>>
>> ** **
>>
>> # Describe the sink ****
>>
>> a1.sinks.k1.type = logger ****
>>
>> # Use a channel which buffers events in memory ****
>>
>> a1.channels.c1.type = memory ****
>>
>> a1.channels.c1.capacity = 1000 ****
>>
>> a1.channels.c1.transactionCapacity = 100 ****
>>
>> ** **
>>
>> # Bind the source and sink to the channel ****
>>
>> a1.sources.r1.channels = c1 ****
>>
>> a1.sinks.k1.channel = c1 ****
>>
>> But I got the following error when run a pig script:
>> java.lang.ClassNotFoundException:
>> org.apache.flume.clients.log4jappender.Log4jAppender
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:169)
>> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
>> at
>> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
>> at
>> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
>> at
>> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
>> at
>> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>> at
>> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
>> at
>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
>> at
>> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
>> at org.apache.pig.Main.configureLog4J(Main.java:678)
>> at org.apache.pig.Main.run(Main.java:337)
>> at org.apache.pig.Main.main(Main.java:111)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Dongliang Sun 2013-01-14, 06:05

REGISTER and -Dpig.additional.libraries both do not work, same error, could
not find the class.

If put the particular jars in pig's lib, the following error:
Error before Pig is launched
----------------------------
ERROR 2998: Unhandled internal error. org/jboss/netty/channel/ChannelFactory

java.lang.NoClassDefFoundError: org/jboss/netty/channel/ChannelFactory
   at
org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:168)
   at
org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:128)
   at
org.apache.flume.clients.log4jappender.Log4jAppender.activateOptions(Log4jAppender.java:184)
   at
org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
   at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
   at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
   at
org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
   at
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
   at
org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
   at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
   at
org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
   at org.apache.pig.Main.configureLog4J(Main.java:678)
   at org.apache.pig.Main.run(Main.java:337)
   at org.apache.pig.Main.main(Main.java:111)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException:
org.jboss.netty.channel.ChannelFactory
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 19 more
2013/1/14 Connor Woodson <[EMAIL PROTECTED]>

> You could try calling 'REGISTER <log4jappender jar>' at the start of your
> script, or on the command line by adding
> -Dpig.additional.libraries=<log4jappender jar> to your Pig call. I don't
> know if that will work or not though.
>
> You could try placing it in pig's lib folder.
>
> You also need to include the flume-ng-sdk jar the same way as the
> log4jappender one, as the sdk is a dependency of the appender.
>
>
> On Sun, Jan 13, 2013 at 9:24 PM, Dongliang Sun <[EMAIL PROTECTED]>wrote:
>
>> Actually I have append the FLUME_HOME/lib/ to my CLASSPATH.
>> Whether there is some problem in my PIG log4j.properties?
>> Do you have any experience using log4jappender on PIG?
>>
>> Thanks,
>> Dongliang
>>
>>
>>
>> 2013/1/14 Chhaya Vishwakarma <[EMAIL PROTECTED]>
>>
>>> Put flume-log4j-appender<https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-log4j-appender/0.9.4-cdh3u1/flume-log4j-appender-0.9.4-cdh3u1.jar>.jar
>>> in CLASSPATH. Jar can be found in FLUME_HOME/lib/****
>>>
>>> ** **
>>>
>>> *From:* 孙东亮 [mailto:[EMAIL PROTECTED]]
>>> *Sent:* Monday, January 14, 2013 8:55 AM
>>> *To:* [EMAIL PROTECTED]
>>> *Subject:* Could not instantiate class
>>> org.apache.flume.clients.log4jappender****
>>>
>>> ** **
>>>
>>> I'm a newbie for the Flume, and I just set up flume for a test.
>>> I want to use the log4jappender to get log info from PIG scripts, the
>>> log4j.properties is:****
>>>
>>> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender ****

RE: Could not instantiate class org.apache.flume.clients.log4jappender

Chhaya Vishwakarma 2013-01-14, 06:10

Hi,
This error is coming because it needs some other jar from FLUME_HOME/lib/ I can’t tell you name of specific jar but I was facing same problem.Put jetty,Jackson,avro,slf4 and some more from FLUME_HOME/lib/ because every time you will run your script will give you different error like java.lang.NoClassDefFoundError

From: Dongliang Sun [mailto:[EMAIL PROTECTED]]
Sent: Monday, January 14, 2013 11:36 AM
To: [EMAIL PROTECTED]; Chhaya Vishwakarma; [EMAIL PROTECTED]
Subject: Re: Could not instantiate class org.apache.flume.clients.log4jappender

REGISTER and -Dpig.additional.libraries both do not work, same error, could not find the class.
If put the particular jars in pig's lib, the following error:
Error before Pig is launched
----------------------------
ERROR 2998: Unhandled internal error. org/jboss/netty/channel/ChannelFactory

java.lang.NoClassDefFoundError: org/jboss/netty/channel/ChannelFactory
   at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:168)
   at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:128)
   at org.apache.flume.clients.log4jappender.Log4jAppender.activateOptions(Log4jAppender.java:184)
   at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
   at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
   at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
   at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
   at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
   at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
   at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
   at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
   at org.apache.pig.Main.configureLog4J(Main.java:678)
   at org.apache.pig.Main.run(Main.java:337)
   at org.apache.pig.Main.main(Main.java:111)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: org.jboss.netty.channel.ChannelFactory
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 19 more

2013/1/14 Connor Woodson <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
You could try calling 'REGISTER <log4jappender jar>' at the start of your script, or on the command line by adding -Dpig.additional.libraries=<log4jappender jar> to your Pig call. I don't know if that will work or not though.

You could try placing it in pig's lib folder.

You also need to include the flume-ng-sdk jar the same way as the log4jappender one, as the sdk is a dependency of the appender.

On Sun, Jan 13, 2013 at 9:24 PM, Dongliang Sun <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Actually I have append the FLUME_HOME/lib/ to my CLASSPATH.
Whether there is some problem in my PIG log4j.properties?
Do you have any experience using log4jappender on PIG?
Thanks,
Dongliang
2013/1/14 Chhaya Vishwakarma <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>

Put flume-log4j-appender<https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-log4j-appender/0.9.4-cdh3u1/flume-log4j-appender-0.9.4-cdh3u1.jar>.jar in CLASSPATH. Jar can be found in FLUME_HOME/lib/

From: 孙东亮 [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Monday, January 14, 2013 8:55 AM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Could not instantiate class org.apache.flume.clients.log4jappender

I'm a newbie for the Flume, and I just set up flume for a test.
I want to use the log4jappender to get log info from PIG scripts, the log4j.properties is:

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender

log4j.appender.flume.Hostname = localhost

log4j.appender.flume.Port = 41414

log4j.logger.org.apache.pig=DEBUG,flume

And for the flume.config:

a1.sources = r1

a1.sinks = k1

a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = avro

a1.sources.r1.bind = localhost

a1.sources.r1.port = 41414

# Describe the sink

a1.sinks.k1.type = logger

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

But I got the following error when run a pig script:
java.lang.ClassNotFoundException: org.apache.flume.clients.log4jappender.Log4jAppender
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
at org.apache.log4j.helpers

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Alexander Alten-Lorenz 2013-01-14, 06:58

Please post the flume startup line. I guess you have missed the --conf switch.

- Alex

On Jan 14, 2013, at 4:25 AM, 孙东亮 <[EMAIL PROTECTED]> wrote:

> I'm a newbie for the Flume, and I just set up flume for a test.
> I want to use the log4jappender to get log info from PIG scripts, the
> log4j.properties is:
>
> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
> log4j.appender.flume.Hostname = localhost
> log4j.appender.flume.Port = 41414
> log4j.logger.org.apache.pig=DEBUG,flume
>
> And for the flume.config:
>
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
>
> # Describe/configure the source
> a1.sources.r1.type = avro
> a1.sources.r1.bind = localhost
> a1.sources.r1.port = 41414
>
> # Describe the sink
> a1.sinks.k1.type = logger
> # Use a channel which buffers events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1000
> a1.channels.c1.transactionCapacity = 100
>
> # Bind the source and sink to the channel
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
>
> But I got the following error when run a pig script:
> java.lang.ClassNotFoundException:
> org.apache.flume.clients.log4jappender.Log4jAppender
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:169)
> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
> at
> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
> at
> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
> at
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
> at
> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
> at org.apache.pig.Main.configureLog4J(Main.java:678)
> at org.apache.pig.Main.run(Main.java:337)
> at org.apache.pig.Main.main(Main.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> log4j:ERROR Could not instantiate appender named "flume".
>
> Could you please help me find what I miss or where is incorrect for the
> configuration.
>
> Thanks a lot!
> Dongliang

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Connor Woodson 2013-01-14, 07:10

He's running Pig, not Flume, but trying to get the Flume Log4jAppender
working.

If you want to try to compile with the log4jappender, download Pig from
Github and add the following to ivy.xml, then 'ant build':

<dependency org="org.apache.flume" name="flume-ng-clients" rev="1.3.1"/>

On Sun, Jan 13, 2013 at 10:58 PM, Alexander Alten-Lorenz <
[EMAIL PROTECTED]> wrote:

> Please post the flume startup line. I guess you have missed the --conf
> switch.
>
> - Alex
>
> On Jan 14, 2013, at 4:25 AM, 孙东亮 <[EMAIL PROTECTED]> wrote:
>
> > I'm a newbie for the Flume, and I just set up flume for a test.
> > I want to use the log4jappender to get log info from PIG scripts, the
> > log4j.properties is:
> >
> > log4j.appender.flume > org.apache.flume.clients.log4jappender.Log4jAppender
> > log4j.appender.flume.Hostname = localhost
> > log4j.appender.flume.Port = 41414
> > log4j.logger.org.apache.pig=DEBUG,flume
> >
> > And for the flume.config:
> >
> > a1.sources = r1
> > a1.sinks = k1
> > a1.channels = c1
> >
> > # Describe/configure the source
> > a1.sources.r1.type = avro
> > a1.sources.r1.bind = localhost
> > a1.sources.r1.port = 41414
> >
> > # Describe the sink
> > a1.sinks.k1.type = logger
> > # Use a channel which buffers events in memory
> > a1.channels.c1.type = memory
> > a1.channels.c1.capacity = 1000
> > a1.channels.c1.transactionCapacity = 100
> >
> > # Bind the source and sink to the channel
> > a1.sources.r1.channels = c1
> > a1.sinks.k1.channel = c1
> >
> > But I got the following error when run a pig script:
> > java.lang.ClassNotFoundException:
> > org.apache.flume.clients.log4jappender.Log4jAppender
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:169)
> > at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
> > at
> >
> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
> > at
> >
> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
> > at
> >
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
> > at
> >
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
> > at
> >
> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
> > at
> >
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
> > at
> >
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
> > at org.apache.pig.Main.configureLog4J(Main.java:678)
> > at org.apache.pig.Main.run(Main.java:337)
> > at org.apache.pig.Main.main(Main.java:111)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > log4j:ERROR Could not instantiate appender named "flume".
> >
> > Could you please help me find what I miss or where is incorrect for the
> > configuration.
> >
> > Thanks a lot!
> > Dongliang
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Alexander Alten-Lorenz 2013-01-14, 08:53

Thanks, thats correct.

@Dongliang: Please follow the instructions Connor gave.

- Alex

On Jan 14, 2013, at 8:10 AM, Connor Woodson <[EMAIL PROTECTED]> wrote:

> He's running Pig, not Flume, but trying to get the Flume Log4jAppender
> working.
>
> If you want to try to compile with the log4jappender, download Pig from
> Github and add the following to ivy.xml, then 'ant build':
>
> <dependency org="org.apache.flume" name="flume-ng-clients" rev="1.3.1"/>
>
>
>
> On Sun, Jan 13, 2013 at 10:58 PM, Alexander Alten-Lorenz <
> [EMAIL PROTECTED]> wrote:
>
>> Please post the flume startup line. I guess you have missed the --conf
>> switch.
>>
>> - Alex
>>
>> On Jan 14, 2013, at 4:25 AM, 孙东亮 <[EMAIL PROTECTED]> wrote:
>>
>>> I'm a newbie for the Flume, and I just set up flume for a test.
>>> I want to use the log4jappender to get log info from PIG scripts, the
>>> log4j.properties is:
>>>
>>> log4j.appender.flume >> org.apache.flume.clients.log4jappender.Log4jAppender
>>> log4j.appender.flume.Hostname = localhost
>>> log4j.appender.flume.Port = 41414
>>> log4j.logger.org.apache.pig=DEBUG,flume
>>>
>>> And for the flume.config:
>>>
>>> a1.sources = r1
>>> a1.sinks = k1
>>> a1.channels = c1
>>>
>>> # Describe/configure the source
>>> a1.sources.r1.type = avro
>>> a1.sources.r1.bind = localhost
>>> a1.sources.r1.port = 41414
>>>
>>> # Describe the sink
>>> a1.sinks.k1.type = logger
>>> # Use a channel which buffers events in memory
>>> a1.channels.c1.type = memory
>>> a1.channels.c1.capacity = 1000
>>> a1.channels.c1.transactionCapacity = 100
>>>
>>> # Bind the source and sink to the channel
>>> a1.sources.r1.channels = c1
>>> a1.sinks.k1.channel = c1
>>>
>>> But I got the following error when run a pig script:
>>> java.lang.ClassNotFoundException:
>>> org.apache.flume.clients.log4jappender.Log4jAppender
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
>>> at
>>>
>> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
>>> at
>>>
>> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
>>> at
>>>
>> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
>>> at
>>>
>> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>>> at
>>>
>> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
>>> at
>>>
>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
>>> at
>>>
>> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
>>> at org.apache.pig.Main.configureLog4J(Main.java:678)
>>> at org.apache.pig.Main.run(Main.java:337)
>>> at org.apache.pig.Main.main(Main.java:111)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> log4j:ERROR Could not instantiate appender named "flume".
>>>
>>> Could you please help me find what I miss or where is incorrect for the
>>> configuration.
>>>
>>> Thanks a lot!
>>> Dongliang
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Dongliang Sun 2013-01-14, 09:30

I have followed Connor's instructions and build the pig from pig-trunk.

But get the same error if I do not do anything else.
log4j:ERROR Could not instantiate class
[org.apache.flume.clients.log4jappender.Log4jAppender].
java.lang.ClassNotFoundException:
org.apache.flume.clients.log4jappender.Log4jAppender

So what should I do next? Copy the jars from $FLUME_HOME/lib to
pig-trunk/lib manually?
2013/1/14 Alexander Alten-Lorenz <[EMAIL PROTECTED]>

> Thanks, thats correct.
>
> @Dongliang: Please follow the instructions Connor gave.
>
> - Alex
>
> On Jan 14, 2013, at 8:10 AM, Connor Woodson <[EMAIL PROTECTED]>
> wrote:
>
> > He's running Pig, not Flume, but trying to get the Flume Log4jAppender
> > working.
> >
> > If you want to try to compile with the log4jappender, download Pig from
> > Github and add the following to ivy.xml, then 'ant build':
> >
> > <dependency org="org.apache.flume" name="flume-ng-clients" rev="1.3.1"/>
> >
> >
> >
> > On Sun, Jan 13, 2013 at 10:58 PM, Alexander Alten-Lorenz <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Please post the flume startup line. I guess you have missed the --conf
> >> switch.
> >>
> >> - Alex
> >>
> >> On Jan 14, 2013, at 4:25 AM, 孙东亮 <[EMAIL PROTECTED]> wrote:
> >>
> >>> I'm a newbie for the Flume, and I just set up flume for a test.
> >>> I want to use the log4jappender to get log info from PIG scripts, the
> >>> log4j.properties is:
> >>>
> >>> log4j.appender.flume > >> org.apache.flume.clients.log4jappender.Log4jAppender
> >>> log4j.appender.flume.Hostname = localhost
> >>> log4j.appender.flume.Port = 41414
> >>> log4j.logger.org.apache.pig=DEBUG,flume
> >>>
> >>> And for the flume.config:
> >>>
> >>> a1.sources = r1
> >>> a1.sinks = k1
> >>> a1.channels = c1
> >>>
> >>> # Describe/configure the source
> >>> a1.sources.r1.type = avro
> >>> a1.sources.r1.bind = localhost
> >>> a1.sources.r1.port = 41414
> >>>
> >>> # Describe the sink
> >>> a1.sinks.k1.type = logger
> >>> # Use a channel which buffers events in memory
> >>> a1.channels.c1.type = memory
> >>> a1.channels.c1.capacity = 1000
> >>> a1.channels.c1.transactionCapacity = 100
> >>>
> >>> # Bind the source and sink to the channel
> >>> a1.sources.r1.channels = c1
> >>> a1.sinks.k1.channel = c1
> >>>
> >>> But I got the following error when run a pig script:
> >>> java.lang.ClassNotFoundException:
> >>> org.apache.flume.clients.log4jappender.Log4jAppender
> >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>> at java.security.AccessController.doPrivileged(Native Method)
> >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>> at java.lang.Class.forName0(Native Method)
> >>> at java.lang.Class.forName(Class.java:169)
> >>> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
> >>> at
> >>>
> >>
> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
> >>> at
> >>>
> >>
> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
> >>> at
> >>>
> >>
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
> >>> at
> >>>
> >>
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
> >>> at
> >>>
> >>
> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
> >>> at
> >>>
> >>
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
> >>> at
> >>>
> >>
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
> >>> at org.apache.pig.Main.configureLog4J(Main.java:678)
> >>> at org.apache.pig.Main.run(Main.java:337)
> >>> at org.apache.pig.Main.main(Main.java:111)
> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>> at
> >>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Alexander Alten-Lorenz 2013-01-14, 10:01

May you ask on the Pig mailing list? This looks like not a Flume issue, and I think the pig knowledge is a bit limited here. The only option as I know so far has Connor described.

- Alex

On Jan 14, 2013, at 10:30 AM, Dongliang Sun <[EMAIL PROTECTED]> wrote:

> I have followed Connor's instructions and build the pig from pig-trunk.
>
> But get the same error if I do not do anything else.
> log4j:ERROR Could not instantiate class
> [org.apache.flume.clients.log4jappender.Log4jAppender].
> java.lang.ClassNotFoundException:
> org.apache.flume.clients.log4jappender.Log4jAppender
>
> So what should I do next? Copy the jars from $FLUME_HOME/lib to
> pig-trunk/lib manually?
>
>
> 2013/1/14 Alexander Alten-Lorenz <[EMAIL PROTECTED]>
>
>> Thanks, thats correct.
>>
>> @Dongliang: Please follow the instructions Connor gave.
>>
>> - Alex
>>
>> On Jan 14, 2013, at 8:10 AM, Connor Woodson <[EMAIL PROTECTED]>
>> wrote:
>>
>>> He's running Pig, not Flume, but trying to get the Flume Log4jAppender
>>> working.
>>>
>>> If you want to try to compile with the log4jappender, download Pig from
>>> Github and add the following to ivy.xml, then 'ant build':
>>>
>>> <dependency org="org.apache.flume" name="flume-ng-clients" rev="1.3.1"/>
>>>
>>>
>>>
>>> On Sun, Jan 13, 2013 at 10:58 PM, Alexander Alten-Lorenz <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Please post the flume startup line. I guess you have missed the --conf
>>>> switch.
>>>>
>>>> - Alex
>>>>
>>>> On Jan 14, 2013, at 4:25 AM, 孙东亮 <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> I'm a newbie for the Flume, and I just set up flume for a test.
>>>>> I want to use the log4jappender to get log info from PIG scripts, the
>>>>> log4j.properties is:
>>>>>
>>>>> log4j.appender.flume >>>> org.apache.flume.clients.log4jappender.Log4jAppender
>>>>> log4j.appender.flume.Hostname = localhost
>>>>> log4j.appender.flume.Port = 41414
>>>>> log4j.logger.org.apache.pig=DEBUG,flume
>>>>>
>>>>> And for the flume.config:
>>>>>
>>>>> a1.sources = r1
>>>>> a1.sinks = k1
>>>>> a1.channels = c1
>>>>>
>>>>> # Describe/configure the source
>>>>> a1.sources.r1.type = avro
>>>>> a1.sources.r1.bind = localhost
>>>>> a1.sources.r1.port = 41414
>>>>>
>>>>> # Describe the sink
>>>>> a1.sinks.k1.type = logger
>>>>> # Use a channel which buffers events in memory
>>>>> a1.channels.c1.type = memory
>>>>> a1.channels.c1.capacity = 1000
>>>>> a1.channels.c1.transactionCapacity = 100
>>>>>
>>>>> # Bind the source and sink to the channel
>>>>> a1.sources.r1.channels = c1
>>>>> a1.sinks.k1.channel = c1
>>>>>
>>>>> But I got the following error when run a pig script:
>>>>> java.lang.ClassNotFoundException:
>>>>> org.apache.flume.clients.log4jappender.Log4jAppender
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>> at java.lang.Class.forName0(Native Method)
>>>>> at java.lang.Class.forName(Class.java:169)
>>>>> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
>>>>> at
>>>>>
>>>>
>> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
>>>>> at
>>>>>
>>>>
>> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
>>>>> at
>>>>>
>>>>
>> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
>>>>> at
>>>>>
>>>>
>> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>>>>> at
>>>>>
>>>>
>> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
>>>>> at
>>>>>
>>>>
>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
>>>>> at
>>>>>
>>>>
>> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)

Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Dongliang Sun 2013-01-14, 07:11

Alex,

The flume startup no problem,
flume-ng agent --conf ./conf/ --conf-file ./conf/flume.conf --name agent1
-Dflume.root.logger=DEBUG,console

I have test the following client,
bin/flume-ng avro-client -H localhost -p 41414 -F ~/test.txt

It's OK, but if I can not make it work if I use the log4jappender as the
client.

Thanks,
Dongliang
2013/1/14 Alexander Alten-Lorenz <[EMAIL PROTECTED]>

> Please post the flume startup line. I guess you have missed the --conf
> switch.
>
> - Alex
>
> On Jan 14, 2013, at 4:25 AM, 孙东亮 <[EMAIL PROTECTED]> wrote:
>
> > I'm a newbie for the Flume, and I just set up flume for a test.
> > I want to use the log4jappender to get log info from PIG scripts, the
> > log4j.properties is:
> >
> > log4j.appender.flume > org.apache.flume.clients.log4jappender.Log4jAppender
> > log4j.appender.flume.Hostname = localhost
> > log4j.appender.flume.Port = 41414
> > log4j.logger.org.apache.pig=DEBUG,flume
> >
> > And for the flume.config:
> >
> > a1.sources = r1
> > a1.sinks = k1
> > a1.channels = c1
> >
> > # Describe/configure the source
> > a1.sources.r1.type = avro
> > a1.sources.r1.bind = localhost
> > a1.sources.r1.port = 41414
> >
> > # Describe the sink
> > a1.sinks.k1.type = logger
> > # Use a channel which buffers events in memory
> > a1.channels.c1.type = memory
> > a1.channels.c1.capacity = 1000
> > a1.channels.c1.transactionCapacity = 100
> >
> > # Bind the source and sink to the channel
> > a1.sources.r1.channels = c1
> > a1.sinks.k1.channel = c1
> >
> > But I got the following error when run a pig script:
> > java.lang.ClassNotFoundException:
> > org.apache.flume.clients.log4jappender.Log4jAppender
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:169)
> > at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
> > at
> >
> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
> > at
> >
> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
> > at
> >
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
> > at
> >
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
> > at
> >
> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
> > at
> >
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
> > at
> >
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
> > at org.apache.pig.Main.configureLog4J(Main.java:678)
> > at org.apache.pig.Main.run(Main.java:337)
> > at org.apache.pig.Main.main(Main.java:111)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > log4j:ERROR Could not instantiate appender named "flume".
> >
> > Could you please help me find what I miss or where is incorrect for the
> > configuration.
> >
> > Thanks a lot!
> > Dongliang
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Alexander Alten-Lorenz 2013-01-14, 08:13

Can you please post the running flume process (ps waux|grep flume)?

- Alex

On Jan 14, 2013, at 8:11 AM, Dongliang Sun <[EMAIL PROTECTED]> wrote:

> Alex,
>
> The flume startup no problem,
> flume-ng agent --conf ./conf/ --conf-file ./conf/flume.conf --name agent1
> -Dflume.root.logger=DEBUG,console
>
> I have test the following client,
> bin/flume-ng avro-client -H localhost -p 41414 -F ~/test.txt
>
> It's OK, but if I can not make it work if I use the log4jappender as the
> client.
>
> Thanks,
> Dongliang
>
>
> 2013/1/14 Alexander Alten-Lorenz <[EMAIL PROTECTED]>
>
>> Please post the flume startup line. I guess you have missed the --conf
>> switch.
>>
>> - Alex
>>
>> On Jan 14, 2013, at 4:25 AM, 孙东亮 <[EMAIL PROTECTED]> wrote:
>>
>>> I'm a newbie for the Flume, and I just set up flume for a test.
>>> I want to use the log4jappender to get log info from PIG scripts, the
>>> log4j.properties is:
>>>
>>> log4j.appender.flume >> org.apache.flume.clients.log4jappender.Log4jAppender
>>> log4j.appender.flume.Hostname = localhost
>>> log4j.appender.flume.Port = 41414
>>> log4j.logger.org.apache.pig=DEBUG,flume
>>>
>>> And for the flume.config:
>>>
>>> a1.sources = r1
>>> a1.sinks = k1
>>> a1.channels = c1
>>>
>>> # Describe/configure the source
>>> a1.sources.r1.type = avro
>>> a1.sources.r1.bind = localhost
>>> a1.sources.r1.port = 41414
>>>
>>> # Describe the sink
>>> a1.sinks.k1.type = logger
>>> # Use a channel which buffers events in memory
>>> a1.channels.c1.type = memory
>>> a1.channels.c1.capacity = 1000
>>> a1.channels.c1.transactionCapacity = 100
>>>
>>> # Bind the source and sink to the channel
>>> a1.sources.r1.channels = c1
>>> a1.sinks.k1.channel = c1
>>>
>>> But I got the following error when run a pig script:
>>> java.lang.ClassNotFoundException:
>>> org.apache.flume.clients.log4jappender.Log4jAppender
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
>>> at
>>>
>> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
>>> at
>>>
>> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
>>> at
>>>
>> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
>>> at
>>>
>> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>>> at
>>>
>> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
>>> at
>>>
>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
>>> at
>>>
>> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
>>> at org.apache.pig.Main.configureLog4J(Main.java:678)
>>> at org.apache.pig.Main.run(Main.java:337)
>>> at org.apache.pig.Main.main(Main.java:111)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> log4j:ERROR Could not instantiate appender named "flume".
>>>
>>> Could you please help me find what I miss or where is incorrect for the
>>> configuration.
>>>
>>> Thanks a lot!
>>> Dongliang
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Dongliang Sun 2013-01-14, 08:37

The result from ps waux | grep flume :
hadoop 10160 2.8 3.2 186672 32960 pts/2 Sl+ 16:35 0:00
/usr/lib/jvm/jdk1.6.0_37/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console
-cp
/home/hadoop/softwares/apache-flume-1.3.1-bin/conf:/home/hadoop/softwares/apache-flume-1.3.1-bin/lib/*:/home/hadoop/softwares/hadoop-1.1.1/libexec/../conf:/usr/lib/jvm/jdk1.6.0_37/lib/tools.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/..:/home/hadoop/softwares/hadoop-1.1.1/libexec/../hadoop-core-1.1.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/asm-3.2.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/aspectjrt-1.6.11.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/aspectjtools-1.6.11.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-beanutils-1.7.0.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-cli-1.2.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-codec-1.4.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-collections-3.2.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-configuration-1.6.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-daemon-1.0.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-digester-1.8.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-el-1.0.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-httpclient-3.0.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-io-2.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-lang-2.4.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-logging-1.1.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-logging-api-1.0.4.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-math-2.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/commons-net-3.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/core-3.1.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/hadoop-capacity-scheduler-1.1.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/hadoop-fairscheduler-1.1.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/hadoop-thriftfs-1.1.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/hsqldb-1.8.0.10.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jasper-compiler-5.5.12.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jasper-runtime-5.5.12.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jdeb-0.8.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jersey-core-1.8.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jersey-json-1.8.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jersey-server-1.8.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jets3t-0.6.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jetty-6.1.26.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jetty-util-6.1.26.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jsch-0.1.42.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/junit-4.5.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/kfs-0.2.2.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/log4j-1.2.15.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/mockito-all-1.8.5.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/oro-2.0.8.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/servlet-api-2.5-20081211.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/xmlenc-0.52.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar
-Djava.library.path=:/home/hadoop/softwares/hadoop-1.1.1/libexec/../lib/native/Linux-i386-32
org.apache.flume.node.Application --conf-file ./conf/flume.conf --name
agent1
2013/1/14 Alexander Alten-Lorenz <[EMAIL PROTECTED]>

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Dongliang Sun 2013-01-14, 05:42

Chhaya,

Thanks for your help. I have followed your instruction, put the particular
jar(flume-ng-log4jappender-1.3.1.jar) in my CLASSPATH also, but it still
does not work, same error.
2013/1/14 Chhaya Vishwakarma <[EMAIL PROTECTED]>

> Place that particular jar also.it will work****
>
> ** **
>
> *From:* Dongliang Sun [mailto:[EMAIL PROTECTED]]
> *Sent:* Monday, January 14, 2013 10:55 AM
> *To:* Chhaya Vishwakarma; [EMAIL PROTECTED]
> *Subject:* Re: Could not instantiate class
> org.apache.flume.clients.log4jappender****
>
> ** **
>
> Actually I have append the FLUME_HOME/lib/ to my CLASSPATH.****
>
> Whether there is some problem in my PIG log4j.properties?****
>
> Do you have any experience using log4jappender on PIG?****
>
> Thanks,****
>
> Dongliang****
>
> ** **
>
> 2013/1/14 Chhaya Vishwakarma <[EMAIL PROTECTED]>****
>
> Put flume-log4j-appender<https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-log4j-appender/0.9.4-cdh3u1/flume-log4j-appender-0.9.4-cdh3u1.jar>.jar
> in CLASSPATH. Jar can be found in FLUME_HOME/lib/****
>
> ****
>
> *From:* 孙东亮 [mailto:[EMAIL PROTECTED]]
> *Sent:* Monday, January 14, 2013 8:55 AM
> *To:* [EMAIL PROTECTED]
> *Subject:* Could not instantiate class
> org.apache.flume.clients.log4jappender****
>
> ****
>
> I'm a newbie for the Flume, and I just set up flume for a test.
> I want to use the log4jappender to get log info from PIG scripts, the
> log4j.properties is:****
>
> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender ****
>
> log4j.appender.flume.Hostname = localhost ****
>
> log4j.appender.flume.Port = 41414 ****
>
> log4j.logger.org.apache.pig=DEBUG,flume****
>
> And for the flume.config: ****
>
> a1.sources = r1 ****
>
> a1.sinks = k1 ****
>
> a1.channels = c1 ****
>
> ****
>
> # Describe/configure the source****
>
> a1.sources.r1.type = avro ****
>
> a1.sources.r1.bind = localhost ****
>
> a1.sources.r1.port = 41414 ****
>
> ****
>
> # Describe the sink ****
>
> a1.sinks.k1.type = logger ****
>
> # Use a channel which buffers events in memory ****
>
> a1.channels.c1.type = memory ****
>
> a1.channels.c1.capacity = 1000 ****
>
> a1.channels.c1.transactionCapacity = 100 ****
>
> ****
>
> # Bind the source and sink to the channel ****
>
> a1.sources.r1.channels = c1 ****
>
> a1.sinks.k1.channel = c1 ****
>
> But I got the following error when run a pig script:
> java.lang.ClassNotFoundException:
> org.apache.flume.clients.log4jappender.Log4jAppender
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:169)
> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
> at
> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
> at
> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
> at
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
> at
> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
> at org.apache.pig.Main.configureLog4J(Main.java:678)
> at org.apache.pig.Main.run(Main.java:337)
> at org.apache.pig.Main.main(Main.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

Re: Could not instantiate class org.apache.flume.clients.log4jappender

Dongliang Sun 2013-01-14, 06:24

It does not work, -Dpig.additional.jars={REALPATH}/lib/*.

Also tried Chhaya said, I put all the libraries from FLUME_HOME/lib/* to
PIG_HOME/lib/, got following error:

13/01/14 14:19:03 WARN api.NettyAvroRpcClient: Invalid value for batchSize:
0; Using default value.
Exception in thread "main" java.lang.NoSuchMethodError:
org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
   at
org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:211)
   at org.apache.pig.impl.util.LogUtils.writeLog(LogUtils.java:125)
   at org.apache.pig.impl.util.LogUtils.writeLog(LogUtils.java:79)
   at org.apache.pig.Main.run(Main.java:589)
   at org.apache.pig.Main.main(Main.java:111)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2013/1/14 Connor Woodson <[EMAIL PROTECTED]>

> Alright so it is just before you weren't adding them properly to the
> classpath. I am not quite sure how to do that; a look at Pig's run script
> (bin/pig) might yield insight into an option to add them. Putting the files
> in pig's lib's folder almost works; the issue is those libraries have more
> dependencies that aren't in the classpath yet. Looking at their
> (log4jappender and sdk) poms could tell you exactly which libs you need to
> include, but let's not do that.
>
> I mistyped the -D command from before. Try this:
>
> -Dpig.additional.jars=${FLUME_HOME}/lib/*
>
> FLUME_HOME obviously needs to be declared beforehand, or just replace that
> with the real path (and make sure the log4jappender and sdk jars, and all
> other jars, are in the libs folder)
>
>
> On Sun, Jan 13, 2013 at 10:05 PM, Dongliang Sun <[EMAIL PROTECTED]>wrote:
>
>> REGISTER and -Dpig.additional.libraries both do not work, same error,
>> could not find the class.
>>
>> If put the particular jars in pig's lib, the following error:
>> Error before Pig is launched
>> ----------------------------
>> ERROR 2998: Unhandled internal error.
>> org/jboss/netty/channel/ChannelFactory
>>
>> java.lang.NoClassDefFoundError: org/jboss/netty/channel/ChannelFactory
>> at
>> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:168)
>> at
>> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:128)
>> at
>> org.apache.flume.clients.log4jappender.Log4jAppender.activateOptions(Log4jAppender.java:184)
>> at
>> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>> at
>> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>> at
>> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>> at
>> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>>
>> at
>> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>> at
>> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
>> at
>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
>> at
>> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
>> at org.apache.pig.Main.configureLog4J(Main.java:678)
>> at org.apache.pig.Main.run(Main.java:337)
>> at org.apache.pig.Main.main(Main.java:111)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)

RE: Could not instantiate class org.apache.flume.clients.log4jappender

Chhaya Vishwakarma 2013-01-14, 07:09

Which version of slf4j you using ? for this error you need to put SLF4J 1.7.2 also in classpath as org.slf4j.spi.LocationAwareLogger.log is available in 1.7.2 of slf4j.

From: Dongliang Sun [mailto:[EMAIL PROTECTED]]
Sent: Monday, January 14, 2013 11:54 AM
To: Connor Woodson; Chhaya Vishwakarma; [EMAIL PROTECTED]
Subject: Re: Could not instantiate class org.apache.flume.clients.log4jappender

It does not work, -Dpig.additional.jars={REALPATH}/lib/*.
Also tried Chhaya said, I put all the libraries from FLUME_HOME/lib/* to PIG_HOME/lib/, got following error:

13/01/14 14:19:03 WARN api.NettyAvroRpcClient: Invalid value for batchSize: 0; Using default value.
Exception in thread "main" java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
   at org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:211)
   at org.apache.pig.impl.util.LogUtils.writeLog(LogUtils.java:125)
   at org.apache.pig.impl.util.LogUtils.writeLog(LogUtils.java:79)
   at org.apache.pig.Main.run(Main.java:589)
   at org.apache.pig.Main.main(Main.java:111)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

2013/1/14 Connor Woodson <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Alright so it is just before you weren't adding them properly to the classpath. I am not quite sure how to do that; a look at Pig's run script (bin/pig) might yield insight into an option to add them. Putting the files in pig's lib's folder almost works; the issue is those libraries have more dependencies that aren't in the classpath yet. Looking at their (log4jappender and sdk) poms could tell you exactly which libs you need to include, but let's not do that.

I mistyped the -D command from before. Try this:

-Dpig.additional.jars=${FLUME_HOME}/lib/*

FLUME_HOME obviously needs to be declared beforehand, or just replace that with the real path (and make sure the log4jappender and sdk jars, and all other jars, are in the libs folder)

On Sun, Jan 13, 2013 at 10:05 PM, Dongliang Sun <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
REGISTER and -Dpig.additional.libraries both do not work, same error, could not find the class.
If put the particular jars in pig's lib, the following error:
Error before Pig is launched
----------------------------
ERROR 2998: Unhandled internal error. org/jboss/netty/channel/ChannelFactory

java.lang.NoClassDefFoundError: org/jboss/netty/channel/ChannelFactory
   at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:168)
   at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:128)
   at org.apache.flume.clients.log4jappender.Log4jAppender.activateOptions(Log4jAppender.java:184)
   at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
   at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
   at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
   at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)

   at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
   at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
   at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
   at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
   at org.apache.pig.Main.configureLog4J(Main.java:678)
   at org.apache.pig.Main.run(Main.java:337)
   at org.apache.pig.Main.main(Main.java:111)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: org.jboss.netty.channel.ChannelFactory

   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 19 more

2013/1/14 Connor Woodson <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
You could try calling 'REGISTER <log4jappender jar>' at the start of your script, or on the command line by adding -Dpig.additional.libraries=<log4jappender jar> to your Pig call. I don't know if that will work or not though.

You could try placing it in pig's lib folder.

You also need to include the flume-ng-sdk jar the same way as the log4jappender one, as the sdk is a dependency of the appender.

On Sun, Jan 13, 2013 at 9:24 PM, Dongliang Sun <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Actually I have append the FLUME_HOME/lib/ to my CLASSPATH.
Whether there is some problem in my PIG log4j.properties?
Do you have any experience using log4jappender on PIG?
Thanks,
Dongliang
2013/1/14 Chhaya Vishwakarma <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>

Put flume-log4j-appender<https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-log4j-appender/0.9.4-cdh3u1/flume-log4j-appender-0.9.4-cdh3u1.ja