-Re: Problem reading HDFS block size > 1GB
Brian Bockelman 2009-09-29, 21:58
It breaks at 2^31 bytes = 2 GB. Any size smaller than that should work.
On Sep 29, 2009, at 4:14 PM, Vinay Setty wrote:
> Hi Owen,
> Thank you for the quick reply. Can you please tell me what is the
> limit with which HDFS is known to work?
> On Tue, Sep 29, 2009 at 8:20 PM, Owen O'Malley <[EMAIL PROTECTED]>
>> On Sep 29, 2009, at 10:59 AM, Vinay Setty wrote:
>> We are running Yahoo distribution of Hadoop based on Hadoop
>>> On a 10 nodes cluster with OpenSUSE Linux Operating System. We
>>> have HDFS
>>> configured with Block Size 5GB (This is for our experiments).
>> There is a known limitation to HDFS to blocks of less than 2^31
>> Fixing it would be tedious and no one has signed up to take a pass
>> at it.
>> -- Owen