Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> how to control (or understand) the memory usage in hdfs


+
Ted 2013-03-23, 04:33
+
Harsh J 2013-03-23, 05:14
+
Ted 2013-03-23, 09:00
+
Harsh J 2013-03-23, 10:35
Copy link to this message
-
Re: how to control (or understand) the memory usage in hdfs
oh, really?

ulimit -n is 2048, I'd assumed that would be sufficient for just
testing on my machine. I was going to use 4096 in production.
my hdfs-site.xml has "dfs.datanode.max.xcievers" set to 4096.

As for my logs... there's a lot of "INFO" entries, I haven't gotten
around to configuring it down yet - I'm not quite sure why it's so
extensive at INFO level. My log files is 4.4gb (is this a sign I've
configured or done something wrong?)

I grep -v "INFO" in the log to get the actual error entry (assuming
the stack trace is actually is on the same line or else those stack
lines maybe misleading)
------------
2013-03-23 15:11:43,653 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(127.0.0.1:50010,
storageID=DS-1419421989-192.168.1.5-50010-1363780956652,
infoPort=50075, ipcPort=50020):DataXceiveServer: Exiting due
to:java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:133)
at java.lang.Thread.run(Thread.java:722)

2013-03-23 15:11:44,177 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(127.0.0.1:50010,
storageID=DS-1419421989-192.168.1.5-50010-1363780956652,
infoPort=50075, ipcPort=50020):DataXceiver
java.io.InterruptedIOException: Interruped while waiting for IO on
channel java.nio.channels.SocketChannel[closed]. 0 millis timeout
left.
at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:349)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:292)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:339)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
at java.lang.Thread.run(Thread.java:722)

On 3/23/13, Harsh J <[EMAIL PROTECTED]> wrote:
> I'm guessing your OutOfMemory then is due to "Unable to create native
> thread" message? Do you mind sharing your error logs with us? Cause if
> its that, then its a ulimit/system limits issue and not a real memory
> issue.
>
> On Sat, Mar 23, 2013 at 2:30 PM, Ted <[EMAIL PROTECTED]> wrote:
>> I just checked and after running my tests, I generate only 670mb of
>> data, on 89 blocks.
>>
>> What's more, when I ran the test this time, I had increased my memory
>> to 2048mb so it completed fine - but I decided to run jconsole through
>> the test so I could see what's happenning. The data node never
>> exceeded 200mb of memory usage. It mostly stayed under 100mb.
>>
>> I'm not sure why it would complain about out of memory and shut itself
>> down when it was only 1024. It was fairly consistently doing that the
>> last few days including this morning right before I switched it to
>> 2048.
>>
>> I'm going to run the test again with 1024mb and jconsole running, none
>> of this makes any sense to me.
>>
>> On 3/23/13, Harsh J <[EMAIL PROTECTED]> wrote:
>>> I run a 128 MB heap size DN for my simple purposes on my Mac and it
>>> runs well for what load I apply on it.
>>>
>>> A DN's primary, growing memory consumption comes from the # of blocks
>>> it carries. All of these blocks' file paths are mapped and kept in the
>>> RAM during its lifetime. If your DN has acquired a lot of blocks by
Ted.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB