Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Hadoop with S3 instead of local storage


+
Alok Kumar 2012-08-02, 07:14
+
Harsh J 2012-08-02, 11:52
+
Alok Kumar 2012-08-02, 13:01
+
Harsh J 2012-08-02, 17:48
+
Alok Kumar 2012-08-03, 07:26
Copy link to this message
-
Re: Hadoop with S3 instead of local storage
Alok,

Caused by: java.io.IOException: createNonRecursive unsupported for
this filesystem class org.apache.hadoop.fs.s3.S3FileSystem

This seems like a limitation imposed by HBase. Can you ask your
question at [EMAIL PROTECTED] for the right people to answer you
back? Also, can you run HBase in standalone mode (no RSes)? I believe
thats how it may work on S3?

On Fri, Aug 3, 2012 at 12:56 PM, Alok Kumar <[EMAIL PROTECTED]> wrote:
> Thank you Harsh.
> That clears my doubt for Hadoop with S3.
>
> Q. Does HBase communicate with S3 directly without using Hadoop?
>
> I've put this task aside for a while..! ..will post again.
> I've not make it working yet. "jets3t jar" is present in classpath.
>
> Thanks,
> Alok
>
>
> HMaster is running ..
>
> Regionserver log :
>
> 2012-08-03 12:42:40,576 WARN
> org.jets3t.service.impl.rest.httpclient.RestS3Service: Response
> '/%2Fhbase%2F.logs%2Fslave-1%2C60020%2C1343977957962' - Unexpected response
> code 404, expected 200
> 2012-08-03 12:42:40,576 WARN
> org.jets3t.service.impl.rest.httpclient.RestS3Service: Response
> '/%2Fhbase%2F.logs%2Fslave-1%2C60020%2C1343977957962' - Received error
> response with XML message
> 2012-08-03 12:42:43,063 WARN
> org.jets3t.service.impl.rest.httpclient.RestS3Service: Response
> '/%2Fhbase%2F.logs%2Fslave-1%2C60020%2C1343977957962' - Unexpected response
> code 404, expected 200
> 2012-08-03 12:42:43,063 WARN
> org.jets3t.service.impl.rest.httpclient.RestS3Service: Response
> '/%2Fhbase%2F.logs%2Fslave-1%2C60020%2C1343977957962' - Received error
> response with XML message
> 2012-08-03 12:42:43,831 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
> HLog configuration: blocksize=32 MB, rollsize=30.4 MB, enabled=true,
> optionallogflushinternal=1000ms
> 2012-08-03 12:42:43,840 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Failed
> initialization
> 2012-08-03 12:42:43,842 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed init
> java.io.IOException: cannot get log writer
>     at
> org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:678)
>     at
> org.apache.hadoop.hbase.regionserver.wal.HLog.createWriterInstance(HLog.java:625)
>     at
> org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:557)
>     at
> org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:517)
>     at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:405)
>     at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:331)
>     at
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateHLog(HRegionServer.java:1215)
>     at
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1204)
>     at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:923)
>     at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:639)
>     at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: java.io.IOException: createNonRecursive
> unsupported for this filesystem class org.apache.hadoop.fs.s3.S3FileSystem
>     at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.init(SequenceFileLogWriter.java:106)
>     at
> org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:675)
>     ... 10 more
> Caused by: java.io.IOException: createNonRecursive unsupported for this
> filesystem class org.apache.hadoop.fs.s3.S3FileSystem
>     at
> org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:626)
>     at
> org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:601)
>     at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:442)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)

Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB