Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - RE: Merging of files back in hadoop

Shreya Chakravarty 2011-05-04, 05:38
Robert Evans 2011-05-04, 16:23
Shreya Chakravarty 2011-04-19, 09:19
Copy link to this message
Re: Merging of files back in hadoop
Robert Evans 2011-04-19, 13:37

The metadata is all stored in the name node.  It stores where all of the block are located and the order of the blocks in a file.   Data is merged as needed behind when you call methods on the instance of the java.io.InputStream returned when calling open.  So, when you open a file for reading you are making a connection to one of the machines that has a copy of the first block of the file.  As you read the data and you finish with the first block the second block is then fetched for you from what ever machine has a copy of it and you continue until all blocks are read.  Typically in map/reduce each mapper, that is reading data will read one block, and possibly a little bit more from the start of the next block.  That way you never have all of the file in memory on any machine.  Typically they only process a small part of the block at a time, one key/value pair.  However there is nothing stopping you from doing something bad, and trying to cache the entire contents of the file in memory as you read it from the stream, except that you would eventually get an out of memory exception.

Bobby Evans

On 4/19/11 4:19 AM, "Shreya Chakravarty" <[EMAIL PROTECTED]> wrote:


I have a query regarding how Hadoop merges the data back which has been split into blocks and stored on different nodes.
*        Where is the data merged as we say that the file can be so huge that it doesn't fit onto one machine

*        Where is the sequence maintained for merging it back.

Thanks and Regards,
Shreya Chakravarty

DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

Ayon Sinha 2011-04-19, 15:53