Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Hadoop with S3 instead of local storage


Copy link to this message
-
Re: Hadoop with S3 instead of local storage
Harsh J 2012-08-03, 07:30
Alok,

Caused by: java.io.IOException: createNonRecursive unsupported for
this filesystem class org.apache.hadoop.fs.s3.S3FileSystem

This seems like a limitation imposed by HBase. Can you ask your
question at [EMAIL PROTECTED] for the right people to answer you
back? Also, can you run HBase in standalone mode (no RSes)? I believe
thats how it may work on S3?

On Fri, Aug 3, 2012 at 12:56 PM, Alok Kumar <[EMAIL PROTECTED]> wrote:
> Thank you Harsh.
> That clears my doubt for Hadoop with S3.
>
> Q. Does HBase communicate with S3 directly without using Hadoop?
>
> I've put this task aside for a while..! ..will post again.
> I've not make it working yet. "jets3t jar" is present in classpath.
>
> Thanks,
> Alok
>
>
> HMaster is running ..
>
> Regionserver log :
>
> 2012-08-03 12:42:40,576 WARN
> org.jets3t.service.impl.rest.httpclient.RestS3Service: Response
> '/%2Fhbase%2F.logs%2Fslave-1%2C60020%2C1343977957962' - Unexpected response
> code 404, expected 200
> 2012-08-03 12:42:40,576 WARN
> org.jets3t.service.impl.rest.httpclient.RestS3Service: Response
> '/%2Fhbase%2F.logs%2Fslave-1%2C60020%2C1343977957962' - Received error
> response with XML message
> 2012-08-03 12:42:43,063 WARN
> org.jets3t.service.impl.rest.httpclient.RestS3Service: Response
> '/%2Fhbase%2F.logs%2Fslave-1%2C60020%2C1343977957962' - Unexpected response
> code 404, expected 200
> 2012-08-03 12:42:43,063 WARN
> org.jets3t.service.impl.rest.httpclient.RestS3Service: Response
> '/%2Fhbase%2F.logs%2Fslave-1%2C60020%2C1343977957962' - Received error
> response with XML message
> 2012-08-03 12:42:43,831 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
> HLog configuration: blocksize=32 MB, rollsize=30.4 MB, enabled=true,
> optionallogflushinternal=1000ms
> 2012-08-03 12:42:43,840 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Failed
> initialization
> 2012-08-03 12:42:43,842 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed init
> java.io.IOException: cannot get log writer
>     at
> org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:678)
>     at
> org.apache.hadoop.hbase.regionserver.wal.HLog.createWriterInstance(HLog.java:625)
>     at
> org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:557)
>     at
> org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:517)
>     at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:405)
>     at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:331)
>     at
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateHLog(HRegionServer.java:1215)
>     at
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1204)
>     at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:923)
>     at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:639)
>     at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: java.io.IOException: createNonRecursive
> unsupported for this filesystem class org.apache.hadoop.fs.s3.S3FileSystem
>     at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.init(SequenceFileLogWriter.java:106)
>     at
> org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:675)
>     ... 10 more
> Caused by: java.io.IOException: createNonRecursive unsupported for this
> filesystem class org.apache.hadoop.fs.s3.S3FileSystem
>     at
> org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:626)
>     at
> org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:601)
>     at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:442)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)

Harsh J