Hi all,

If I want have get a  inputstream in a datanode (and this should contains
exactly 64MB data which I want to replace the content of an existing data
block to store that content), can anyone tell me which is the easiest way
to do this? Moreover, I would like to write this stream in the local
datanode.

Thanks,

Sorry, I found the previous one is full of typo and hard to understand. I
will clarify my question:

If I have got a inputstream in a datanode (and this stream contains exactly
64MB's data which I want to store it as a data block of the HDFS in that
datanode. More specifically, I want to use this block to replace another
exsiting block of one file that already stored in the HDFS), can anyone
tell me which is the easiest way to do this?

Thanks,
On Thu, Nov 8, 2012 at 5:15 PM, Vivi Lang <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> If I want have get a  inputstream in a datanode (and this should contains
> exactly 64MB data which I want to replace the content of an existing data
> block to store that content), can anyone tell me which is the easiest way
> to do this? Moreover, I would like to write this stream in the local
> datanode.
>
> Thanks,
>

Hi Vivi,

I don't think HDFS offers any such API to do this today. What is your
use-case for needing this (replacing a given existing block with
another, arbitrary block?) though?

On Fri, Nov 9, 2012 at 7:19 AM, Vivi Lang <[EMAIL PROTECTED]> wrote:
> Sorry, I found the previous one is full of typo and hard to understand. I
> will clarify my question:
>
> If I have got a inputstream in a datanode (and this stream contains exactly
> 64MB's data which I want to store it as a data block of the HDFS in that
> datanode. More specifically, I want to use this block to replace another
> exsiting block of one file that already stored in the HDFS), can anyone
> tell me which is the easiest way to do this?
>
> Thanks,
>
>
> On Thu, Nov 8, 2012 at 5:15 PM, Vivi Lang <[EMAIL PROTECTED]> wrote:
>
>> Hi all,
>>
>> If I want have get a  inputstream in a datanode (and this should contains
>> exactly 64MB data which I want to replace the content of an existing data
>> block to store that content), can anyone tell me which is the easiest way
>> to do this? Moreover, I would like to write this stream in the local
>> datanode.
>>
>> Thanks,
>>

--
Harsh J

thxs for the replying,
what i want to do is, for example, I have a file which already in hdfs,
this file have lots of blocks, but I know some parts( and the related
offset ) of this file is corrupted and I want to rewrite only a few blocks
(most likely only one block) instead of all, I am wondering is there any
convenient way to do this replacement

Thanks again!

On Friday, November 9, 2012, Harsh J wrote:

> Hi Vivi,
>
> I don't think HDFS offers any such API to do this today. What is your
> use-case for needing this (replacing a given existing block with
> another, arbitrary block?) though?
>
> On Fri, Nov 9, 2012 at 7:19 AM, Vivi Lang <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> > Sorry, I found the previous one is full of typo and hard to understand. I
> > will clarify my question:
> >
> > If I have got a inputstream in a datanode (and this stream contains
> exactly
> > 64MB's data which I want to store it as a data block of the HDFS in that
> > datanode. More specifically, I want to use this block to replace another
> > exsiting block of one file that already stored in the HDFS), can anyone
> > tell me which is the easiest way to do this?
> >
> > Thanks,
> >
> >
> > On Thu, Nov 8, 2012 at 5:15 PM, Vivi Lang <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> >
> >> Hi all,
> >>
> >> If I want have get a  inputstream in a datanode (and this should
> contains
> >> exactly 64MB data which I want to replace the content of an existing
> data
> >> block to store that content), can anyone tell me which is the easiest
> way
> >> to do this? Moreover, I would like to write this stream in the local
> >> datanode.
> >>
> >> Thanks,
> >>
>
>
>
> --
> Harsh J
>

On 9 November 2012 17:37, Vivi Lang <[EMAIL PROTECTED]> wrote:

> thxs for the replying,
> what i want to do is, for example, I have a file which already in hdfs,
> this file have lots of blocks, but I know some parts( and the related
> offset ) of this file is corrupted and I want to rewrite only a few blocks
> (most likely only one block) instead of all, I am wondering is there any
> convenient way to do this replacement
>
> Thanks again!
>
>
Is the situation here that you have corrupted HDFS blocks?

Yes, I know the which block is corrupted and want to rewrite the content of
the block. (asuming the replication factor of the related file is 1)

On Fri, Nov 9, 2012 at 12:33 PM, Steve Loughran <[EMAIL PROTECTED]>wrote:

> On 9 November 2012 17:37, Vivi Lang <[EMAIL PROTECTED]> wrote:
>
> > thxs for the replying,
> > what i want to do is, for example, I have a file which already in hdfs,
> > this file have lots of blocks, but I know some parts( and the related
> > offset ) of this file is corrupted and I want to rewrite only a few
> blocks
> > (most likely only one block) instead of all, I am wondering is there any
> > convenient way to do this replacement
> >
> > Thanks again!
> >
> >
> Is the situation here that you have corrupted HDFS blocks?
>

If you just need some experiment setup,
just set dfs.replication to 1, and using dfsclient on the same datenode to
write data, the block is guaranteed to be the same datanode.
On Fri, Nov 9, 2012 at 7:13 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hi Vivi,
>
> I don't think HDFS offers any such API to do this today. What is your
> use-case for needing this (replacing a given existing block with
> another, arbitrary block?) though?
>
> On Fri, Nov 9, 2012 at 7:19 AM, Vivi Lang <[EMAIL PROTECTED]> wrote:
> > Sorry, I found the previous one is full of typo and hard to understand. I
> > will clarify my question:
> >
> > If I have got a inputstream in a datanode (and this stream contains
> exactly
> > 64MB's data which I want to store it as a data block of the HDFS in that
> > datanode. More specifically, I want to use this block to replace another
> > exsiting block of one file that already stored in the HDFS), can anyone
> > tell me which is the easiest way to do this?
> >
> > Thanks,
> >
> >
> > On Thu, Nov 8, 2012 at 5:15 PM, Vivi Lang <[EMAIL PROTECTED]> wrote:
> >
> >> Hi all,
> >>
> >> If I want have get a  inputstream in a datanode (and this should
> contains
> >> exactly 64MB data which I want to replace the content of an existing
> data
> >> block to store that content), can anyone tell me which is the easiest
> way
> >> to do this? Moreover, I would like to write this stream in the local
> >> datanode.
> >>
> >> Thanks,
> >>
>
>
>
> --
> Harsh J
>

On 9 November 2012 13:29, Binglin Chang <[EMAIL PROTECTED]> wrote:

> If you just need some experiment setup,
> just set dfs.replication to 1, and using dfsclient on the same datenode to
> write data, the block is guaranteed to be the same datanode.
>

There's no guarantees: if there is no space on the local DN it will be
moved off-server.