Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> how can I let camus etl point to hdfs instead of local?


Copy link to this message
-
Re: how can I let camus etl point to hdfs instead of local?
I'm sorry I'm not too familiar with the camus code base. Have you tried
pinging [EMAIL PROTECTED] ?

Thanks,
Neha
On Tue, Sep 3, 2013 at 1:40 PM, Yang <[EMAIL PROTECTED]> wrote:

> I tried to run camus with the properties file in the examples dir:
>
> java -cp ...... com.linkedin.camus.etl.kafka.CamusJob -P
> myproperties.properties
>
>
> then it says that my output dir does not exist:
> ~/tools/camus/camus-etl-kafka$ java -cp
> target/camus-etl-kafka-0.1.0-SNAPSHOT.jar
> com.linkedin.camus.etl.kafka.CamusJob -P camus.properties
> Starting Kafka ETL Job
> The blacklisted topics: []
> The whitelisted topics: []
> Dir Destination set to: /camus/out
> Getting the base paths.
> The execution base path does not exist. Creating the directory
> The history base path does not exist. Creating the directory.
> Exception in thread "main" java.io.FileNotFoundException: File /camus/exec
> does not exist.
> at
>
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
>  at
>
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:801)
>  at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:223)
> at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:556)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>
>
>
> I tried changing the dir to hdfs://localhost/camus/out   it says:
> $ java -cp target/camus-etl-kafka-0.1.0-SNAPSHOT.jar
> com.linkedin.camus.etl.kafka.CamusJob -P camus.properties
> Starting Kafka ETL Job
> The blacklisted topics: []
> The whitelisted topics: []
> Dir Destination set to: hdfs://localhost/camus/out
> Getting the base paths.
> Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
> hdfs://localhost/camus/exec, expected: file:///
>  at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
> at
>
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
>  at
> org.apache.hadoop.fs.LocalFileSystem.pathToFile(LocalFileSystem.java:61)
> at org.apache.hadoop.fs.LocalFileSystem.exists(LocalFileSystem.java:51)
>  at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:211)
> at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:556)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>
>
> so how can I let Camus know that this is in a hdfs environment, and the
> code in CamusJob.java:140
>
>     private Job createJob(Properties props) throws IOException {
>         Job job = new Job(getConf());
>
> get a conf that points to a hdfs setup?
>
> I already set my env HADOOP_CONF_DIR to my running hadoop
>
>
> thanks
> Yang
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB