Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> how to control (or understand) the memory usage in hdfs

Ted 2013-03-23, 04:33
Harsh J 2013-03-23, 05:14
Ted 2013-03-23, 09:00
Harsh J 2013-03-23, 10:35
Copy link to this message
Re: how to control (or understand) the memory usage in hdfs
oh, really?

ulimit -n is 2048, I'd assumed that would be sufficient for just
testing on my machine. I was going to use 4096 in production.
my hdfs-site.xml has "dfs.datanode.max.xcievers" set to 4096.

As for my logs... there's a lot of "INFO" entries, I haven't gotten
around to configuring it down yet - I'm not quite sure why it's so
extensive at INFO level. My log files is 4.4gb (is this a sign I've
configured or done something wrong?)

I grep -v "INFO" in the log to get the actual error entry (assuming
the stack trace is actually is on the same line or else those stack
lines maybe misleading)
2013-03-23 15:11:43,653 ERROR
infoPort=50075, ipcPort=50020):DataXceiveServer: Exiting due
to:java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:133)
at java.lang.Thread.run(Thread.java:722)

2013-03-23 15:11:44,177 ERROR
infoPort=50075, ipcPort=50020):DataXceiver
java.io.InterruptedIOException: Interruped while waiting for IO on
channel java.nio.channels.SocketChannel[closed]. 0 millis timeout
at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:349)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:292)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:339)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
at java.lang.Thread.run(Thread.java:722)

On 3/23/13, Harsh J <[EMAIL PROTECTED]> wrote:
> I'm guessing your OutOfMemory then is due to "Unable to create native
> thread" message? Do you mind sharing your error logs with us? Cause if
> its that, then its a ulimit/system limits issue and not a real memory
> issue.
> On Sat, Mar 23, 2013 at 2:30 PM, Ted <[EMAIL PROTECTED]> wrote:
>> I just checked and after running my tests, I generate only 670mb of
>> data, on 89 blocks.
>> What's more, when I ran the test this time, I had increased my memory
>> to 2048mb so it completed fine - but I decided to run jconsole through
>> the test so I could see what's happenning. The data node never
>> exceeded 200mb of memory usage. It mostly stayed under 100mb.
>> I'm not sure why it would complain about out of memory and shut itself
>> down when it was only 1024. It was fairly consistently doing that the
>> last few days including this morning right before I switched it to
>> 2048.
>> I'm going to run the test again with 1024mb and jconsole running, none
>> of this makes any sense to me.
>> On 3/23/13, Harsh J <[EMAIL PROTECTED]> wrote:
>>> I run a 128 MB heap size DN for my simple purposes on my Mac and it
>>> runs well for what load I apply on it.
>>> A DN's primary, growing memory consumption comes from the # of blocks
>>> it carries. All of these blocks' file paths are mapped and kept in the
>>> RAM during its lifetime. If your DN has acquired a lot of blocks by