Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: hadoop fs -text OutOfMemoryError


Copy link to this message
-
Re: hadoop fs -text OutOfMemoryError
Hi,

What is the value of HADOOP_CLIENT_OPTS in you hadoop-env.sh file?

We had similar problems with running OOM with hadoop fs command (I do not
remember if they were exactly related to -text + snappy), when we decreased
the heap to some small value. With higher value e.g. 1 or 2 GB, we were
fine:

# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="-Xmx2048m ${HADOOP_CLIENT_OPTS}"
2013/12/13 xiao li <[EMAIL PROTECTED]>

> Hi Tao
>
> Thanks for your reply,
>
> This is the code, it is pretty simple.
>
> '
>                     fsDataOutputStream.write(Snappy.compress(NEWLINE));
>                     fsDataOutputStream
> .write(Snappy.compress(json.getBytes("UTF-8")));'
>
>
> but FSDataOutputStream is actually opened for appending, I guess the I
> can't simply append to the snappy file(know nothing about it.)
>
>
>
> ------------------------------
> Date: Fri, 13 Dec 2013 21:42:38 +0800
> Subject: Re: hadoop fs -text OutOfMemoryError
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
>
> can you describe your problems in more details, for example, was snappy
> library installed correctly in your cluster, how did you code yout files
> with snappy, was your file correctly coded with snappy ?
>
>
> 2013/12/13 xiao li <[EMAIL PROTECTED]>
>
> I could view the snappy file with hadoop fs -cat but when i issue the
> -text, it gives me this error though the file size is really tiny. what
> have i done wrong? Thanks
>
> hadoop fs -text /test/SinkToHDFS-ip-.us-west-2.compute.internal-6703-22-
> 20131212-0.snappy
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>  at org.apache.hadoop.io.compress.BlockDecompressorStream.
> getCompressedData(BlockDecompressorStream.java:115)
>  at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(
> BlockDecompressorStream.java:95)
>  at org.apache.hadoop.io.compress.DecompressorStream.read(
> DecompressorStream.java:83)
>  at java.io.InputStream.read(InputStream.java:82)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:78)
>  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
>  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
> at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:86)
>  at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:81)
>  at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:306)
> at org.apache.hadoop.fs.shell.Command.processPathArgument(
> Command.java:278)
>  at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
>  at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
> at org.apache.hadoop.fs.shell.Command.processRawArguments(
> Command.java:190)
>  at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
>  at org.apache.hadoop.fs.FsShell.run(FsShell.java:254)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>  at org.apache.hadoop.fs.FsShell.main(FsShell.java:304)
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB