Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> setting collector on a remote machine

Copy link to this message
Re: setting collector on a remote machine
Yes, you can run the collector on any host with network access to
HDFS. Verify that HADOOP_HOME and HADOOP_CONF_DIR are properly set in
chukwa-env.sh. Also verify that you can connect to hadoop from the
command line with those environment variables set.

On Tuesday, September 7, 2010, Shreya Chakravarty
> Hi,
> We want to setup a collector on a non HDFS machine
> Is that possible? Will the collector be able to send data to hdfs, we tried setting the writer.hdfs.filesystem property to point to the hdfs master, but getting the following error:
> 2010-09-07 05:53:11,153 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/chukwa-0.4.0/conf/
> 2010-09-07 05:53:12,721 INFO main root - initing servletCollector
> 2010-09-07 05:53:12,727 INFO main PipelineStageWriter - using pipelined writers, pipe length is 2
> 2010-09-07 05:53:12,751 INFO Thread-9 SocketTeeWriter - listen thread started
> 2010-09-07 05:53:12,755 INFO main SeqFileWriter - rotateInterval is 300000
> 2010-09-07 05:53:12,755 INFO main SeqFileWriter - outputDir is /chukwa/logs/
> 2010-09-07 05:53:12,755 INFO main SeqFileWriter - fsname is iaas-1-0-16-3e-b7-d1-29:54310
> 2010-09-07 05:53:12,756 INFO main SeqFileWriter - filesystem type from core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2010-09-07 05:53:12,860 ERROR main SeqFileWriter - can't connect to HDFS, trying default file system instead (likely to be local)
> java.io.IOException: No FileSystem for scheme: iaas-1-0-16-3e-b7-d1-29
>         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1375)
>         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
>         at org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
>         at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
>         at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
>         at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
>         at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
>         at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
>         at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
>         at org.mortbay.jetty.Server.doStart(Server.java:224)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121)
> Regards
> Shreya
> =========> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.