Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> Re: Can I move block data directly?


Copy link to this message
-
Re: Can I move block data directly?
I know nothing.

It seems that circumventing normal operations could be very bad.  There was
an example of something similar at hadoop summit.  Some very experienced
contributors decided they should edit meta data..... they broke their
cluster.

Just say no !  ;)
On Jul 8, 2013 9:01 PM, "Azuryy Yu" <[EMAIL PROTECTED]> wrote:

> Hi Harsh,
>
> I also do agree with you that this is crude. and balancer is the right way.
> I just want to slove the problem very quickly. and only a few nodes
> involved.
>
>
> Thanks.
>
>
>
>
> On Tue, Jul 9, 2013 at 8:50 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>
> > Eitan,
> >
> > The block to host mapping isn't persisted in the metadata. This is
> > also the reason why the steps include a restart, which will re-trigger
> > a block report (and avoid gotchas) that will update the NN of the new
> > listing at each DN. Thats what makes this method "crude" at the same
> > time - you're leveraging a behavior thats not guaranteed to be
> > unchanged in future.
> >
> > The balancer is the right way to go about it.
> >
> > On Mon, Jul 8, 2013 at 6:53 PM, Eitan Rosenfeld <[EMAIL PROTECTED]>
> wrote:
> > > Hi Azurry, I'd also like to be able to manually move blocks.
> > >
> > > One piece that is missing in your current approach is updating any
> > > block mappings that the cluster relies on.
> > > The namenode has a mapping of blocks to datanodes, and the datanode
> > > has, as the comments say, a "block -> stream of bytes" mapping.
> > >
> > > As I understand it, the namenode's mappings have to be updated to
> > > reflect the new block locations.
> > > The datanode might not need intervention, I'm not sure.
> > >
> > > Can anyone else chime in on those areas?
> > >
> > > The balancer that Allan suggested likely demonstrates all of the ins
> > > and outs in order successfully complete a block transfer.
> > > Thus, the balancer is where I'll begin my efforts to learn how to
> > > manually move blocks.
> > >
> > > Any other pointers would be helpful.
> > >
> > > Thank you,
> > > Eitan
> > >
> > > On Mon, Jul 8, 2013 at 2:15 PM, Allan <[EMAIL PROTECTED]> wrote:
> > >> If the imbalance is across data nodes then you need to run the
> balancer.
> > >>
> > >> Sent from my iPad
> > >>
> > >> On Jul 8, 2013, at 1:15 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
> > >>
> > >>> Hi Dear all,
> > >>>
> > >>> There are some unbalanced data nodes in my cluster, some nodes
> reached
> > more
> > >>> than 95% disk usage.
> > >>>
> > >>> so Can I move some block data from one node to another node directly?
> > >>>
> > >>> such as: from n1 to n2:
> > >>>
> > >>> 1) scp /data/xxxx/blk_*   n2:/data/subdir11/
> > >>> 2) rm -rf data/xxxx/blk_*
> > >>> 3) hadoop-dameon.sh stop datanode (on n1)
> > >>> 4) hadoop-damon.sh start datanode(on n1)
> > >>> 5) hadoop-dameon.sh stop datanode (on n2)
> > >>> 6) hadoop-damon.sh start datanode(on n2)
> > >>>
> > >>> Am I right? Thanks for any inputs.
> >
> >
> >
> > --
> > Harsh J
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB