Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Data loss on EMR cluster running Hadoop and Hive


Copy link to this message
-
Re: Data loss on EMR cluster running Hadoop and Hive
Max,
Yes, you will get better performance if your data is on HDFS (local/ephemeral) versus S3.

I'm not sure why you couldn't see the bad block.
Next time this happens, try running an hadoop fsck from the name node.

The reason why I was suggesting that you run against S3 is that while slower, its still faster than trying to copy the data to the local disk, run the job and then push the results to S3.

Again, I would suggest that you try and contact support from AWS.

HTH

-Mike

On Sep 4, 2012, at 12:08 PM, Max Hansmire <[EMAIL PROTECTED]> wrote:

> Especially where I am reading from from the file using a Map-Reduce
> job in the next step I am not sure that it makes sense in terms of
> performance to put the file on S3. I have not tested, but my suspicion
> is that the local disk reads on HDFS would outperform reading and
> writing the file to S3.
>
> This is a bad block on HDFS and not the underlying filesystem. I
> thought that HDFS was supposed to be tolerant of native file system
> failures.
>
> Max
>
> On Tue, Sep 4, 2012 at 12:43 PM, Michael Segel
> <[EMAIL PROTECTED]> wrote:
>> Next time, try reading and writing to S3 directly from your hive job.
>>
>> Not sure why the block was bad... What did the AWS folks have to say?
>>
>> -Mike
>>
>> On Sep 4, 2012, at 11:30 AM, Max Hansmire <[EMAIL PROTECTED]> wrote:
>>
>>> I ran into an issue yesterday where one of the blocks on HDFS seems to
>>> have gone away. I would appreciate any help that you can provide.
>>>
>>> I am running Hadoop on Amazon's Elastic Map Reduce (EMR). I am running
>>> hadoop version 0.20.205 and hive version 0.8.1.
>>>
>>> I have a hive table that is written out in the reduce step of a map
>>> reduce job created by hive. This step completed with no errors, but
>>> the next map-reduce job that tries to read it failed with the
>>> following error message.
>>>
>>> "Caused by: java.io.IOException: No live nodes contain current block"
>>>
>>> I ran hadoop fs -cat on the same file and got the same error.
>>>
>>> Looking more closely at the data and name node logs, I see this error
>>> for the same problem block. It is in the name node when trying to read
>>> the data.
>>>
>>> 2012-09-03 11:56:05,054 WARN
>>> org.apache.hadoop.hdfs.server.datanode.DataNode
>>> (org.apache.hadoop.hdfs.server.datanode.DataXceiver@4a7cdff0):
>>> DatanodeRegistration(10.193.39.159:9200,
>>> storageID=DS-2147477684-10.193.39.159-9200-1346659207926,
>>> infoPort=9102, ipcPort=9201):sendBlock() :  Offset 134217727 and
>>> length 1 don't match block blk_-7100869813617535842_5426 ( blockLen
>>> 120152064 )
>>> 2012-09-03 11:56:05,054 WARN
>>> org.apache.hadoop.hdfs.server.datanode.DataNode
>>> (org.apache.hadoop.hdfs.server.datanode.DataXceiver@4a7cdff0):
>>> DatanodeRegistration(10.193.39.159:9200,
>>> storageID=DS-2147477684-10.193.39.159-9200-1346659207926,
>>> infoPort=9102, ipcPort=9201):Got exception while serving
>>> blk_-7100869813617535842_5426 to /10.96.57.112:
>>> java.io.IOException:  Offset 134217727 and length 1 don't match block
>>> blk_-7100869813617535842_5426 ( blockLen 120152064 )
>>>      at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:141)
>>>      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:189)
>>>      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
>>>      at java.lang.Thread.run(Thread.java:662)
>>>
>>> 2012-09-03 11:56:05,054 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode
>>> (org.apache.hadoop.hdfs.server.datanode.DataXceiver@4a7cdff0):
>>> DatanodeRegistration(10.193.39.159:9200,
>>> storageID=DS-2147477684-10.193.39.159-9200-1346659207926,
>>> infoPort=9102, ipcPort=9201):DataXceiver
>>> java.io.IOException:  Offset 134217727 and length 1 don't match block
>>> blk_-7100869813617535842_5426 ( blockLen 120152064 )
>>>      at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:141)
>>>      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:189)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB