Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Writing small files to one big file in hdfs

Copy link to this message
Re: Writing small files to one big file in hdfs
       Rather than just appending the content into a normal text file or
so, you can create a sequence file with the individual smaller file content
as values.


On Tue, Feb 21, 2012 at 10:45 PM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:

> We have small xml files. Currently I am planning to append these small
> files to one file in hdfs so that I can take advantage of splits, larger
> blocks and sequential IO. What I am unsure is if it's ok to append one file
> at a time to this hdfs file
> Could someone suggest if this is ok? Would like to know how other do it.