Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> HDFS single datanode cluster issues


Copy link to this message
-
Re: HDFS single datanode cluster issues
First of all, HDFS isn't really the right choice for single-node
environments.  I would recommend using LocalFileSystem in this case.
If you're evaluating HDFS and only have one computer, it will really
be better to run several VMs to see how it works, rather than running
just one Datanode.

You are correct that there are some issues with the pipeline recovery
code on small clusters.  In a lot of cases, pipeline recovery can make
the whole output stream fail, when it is unable to find enough nodes.
We filed HDFS-5131 to address those issues.

In the meantime, you can set
dfs.client.block.write.replace-datanode-on-failure.enable to false,
and dfs.client.block.write.replace-datanode-on-failure.policy to
NEVER.  Based on your comment, it seems like you are already trying to
do this.  Make sure you are setting this configuration on the client
side as well as on the server side-- then you should not see error
messages about pipeline recovery.

best,
Colin

On Wed, Oct 30, 2013 at 8:06 AM, David Mankellow
<[EMAIL PROTECTED]> wrote:
> We are mapping a 1:1 replication.
>
> We have tried setting
> dfs.client.block.write.replace-datanode-on-failure.enable to NEVER but it
> seems to be ignored.
>
> We have tried the following:
> ==>   <property>
> <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
>     <value>false</value>
>   </property>
>   <property>
> <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
>     <value>NEVER</value>
>   </property>
> ==>   <property>
> <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
>     <value>true</value>
>   </property>
>   <property>
> <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
>     <value>NEVER</value>
>   </property>
> ==>   <property>
> <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
>     <value>true</value>
>   </property>
>
> Any more help would be greatly appreciated.
>
> Thanks,
> Dave
>
>
>
>
> On 30/10/2013 10:50, Allan Wilson wrote:
>>
>> Hi David
>>
>> How does your block replica count compare to the number of datanodes in
>> your cluster?
>>
>> Anyway...I found this in the online doc.  You may want to use the NEVER
>> policy.
>>
>> dfs.client.block.write.replace-datanode-on-failure.enable true If there is
>> a datanode/network failure in the write pipeline, DFSClient will try to
>> remove the failed datanode from the pipeline and then continue writing
>> with
>> the remaining datanodes. As a result, the number of datanodes in the
>> pipeline is decreased. The feature is to add new datanodes to the
>> pipeline.
>> This is a site-wide property to enable/disable the feature. When the
>> cluster size is extremely small, e.g. 3 nodes or less, cluster
>> administrators may want to set the policy to NEVER in the default
>> configuration file or disable this feature. Otherwise, users may
>> experience
>> an unusually high rate of pipeline failures since it is impossible to find
>> new datanodes for replacement. See also
>> dfs.client.block.write.replace-datanode-on-failure.policy
>> dfs.client.block.write.replace-datanode-on-failure.policy DEFAULT This
>> property is used only if the value of
>> dfs.client.block.write.replace-datanode-on-failure.enable is true. ALWAYS:
>> always add a new datanode when an existing datanode is removed. NEVER:
>> never add a new datanode. DEFAULT: Let r be the replication number. Let n
>> be the number of existing datanodes. Add a new datanode only if r is
>> greater than or equal to 3 and either (1) floor(r/2) is greater than or
>> equal to n; or (2) r is greater than n and the block is hflushed/appended.
>>
>> Allan
>> On Oct 30, 2013 5:52 AM, "David Mankellow" <[EMAIL PROTECTED]>
>> wrote:
>>
>>> Hi all,
>>>
>>> We are having difficulty writing any logs to a HDFS cluster of less than
>>> 3
>>> nodes. This has been since the update between cdh4.2 and 4.3 (4.4 is also
>>> the same). Has anything changed that may make this occur and is there
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB