Hey guys,

Sorry for the typo in my last message.I have corrected it.

I would like to perform some additional processing on the data which is
streamed to DFSClient. To my knowledge the class DFSInputStream manages the
stream operations on the client side whenever a file is being read, but i
don't know which class should be modified to add this additional processing
capability to data node. Please clarify.
Thanks in advance

Hi Shesha,

 Take a look at  org.apache.hadoop.hdfs.server.datanode.BlockSender.java

Regards,
Uma
________________________________
From: Sesha Kumar [[EMAIL PROTECTED]]
Sent: Monday, January 16, 2012 7:50 PM
To: [EMAIL PROTECTED]
Subject: Data processing in DFSClient

Hey guys,

Sorry for the typo in my last message.I have corrected it.

I would like to perform some additional processing on the data which is streamed to DFSClient. To my knowledge the class DFSInputStream manages the stream operations on the client side whenever a file is being read, but i don't know which class should be modified to add this additional processing capability to data node. Please clarify.
Thanks in advance

Sesha,

What kind of processing are you attempting to do? Maybe it makes more sense
to just implement a MapReduce job rather than modifying the datanodes?

-Joey

On Mon, Jan 16, 2012 at 9:20 AM, Sesha Kumar <[EMAIL PROTECTED]> wrote:

> Hey guys,
>
> Sorry for the typo in my last message.I have corrected it.
>
> I would like to perform some additional processing on the data which is
> streamed to DFSClient. To my knowledge the class DFSInputStream manages the
> stream operations on the client side whenever a file is being read, but i
> don't know which class should be modified to add this additional processing
> capability to data node. Please clarify.
>
>
> Thanks in advance
>

--
Joseph Echeverria
Cloudera, Inc.
443.305.9434

Sorry for the delay. I'm trying to implement an IEEE paper which combines a
bunch of files into a single file and when the file is requested the
datanode extracts the desired file from the block and sends the file to
DFSClient.

I did not get your complete idea here but before that, file will be splitted into blocks. Then how will you extract ?
Why con't you create tar file and one index file which will maintain index of each file..something like this?

________________________________
From: Sesha Kumar [[EMAIL PROTECTED]]
Sent: Wednesday, January 18, 2012 8:24 PM
To: [EMAIL PROTECTED]
Subject: Re: Data processing in DFSClient
Sorry for the delay. I'm trying to implement an IEEE paper which combines a bunch of files into a single file and when the file is requested the datanode extracts the desired file from the block and sends the file to DFSClient.

I'm currently working on this paper where we try to improve the
performance of HDFS by combining small files into a single file (like HAR),
but this merged file contains at the beginning of each block an index file
which is similar to HAR index file. Datanode uses this index file to obtain
the small file from the block. We use some metadata to find out the block
containing the desired file in the merged file.

Personally I would just use Har :) It sounds like an interesting
project. You might find this document helpful:

http://kazman.shidler.hawaii.edu/ArchDoc.html

It was designed to help contributors navigate the HDFS source tree.

-Joey

On Thu, Jan 19, 2012 at 11:52 AM, Sesha Kumar <[EMAIL PROTECTED]> wrote:
>  I'm currently working on this paper where we try to improve the performance
> of HDFS by combining small files into a single file (like HAR), but this
> merged file contains at the beginning of each block an index file which is
> similar to HAR index file. Datanode uses this index file to obtain the small
> file from the block. We use some metadata to find out the block containing
> the desired file in the merged file.

--
Joseph Echeverria
Cloudera, Inc.
443.305.9434