Hi everyone,

I would like to simulate network delay on 1 node in my cluster, perhaps by
putting the thread to sleep every time it transfers data non-locally. I'm
looking at the source but am not sure where to place the code. Is there a
better way to do it... a tool perhaps? Or could someone point me in the
right direction?

Cheers,

Yipeng

Why would you like to *simulate* network delay? I haven't got your point,

Bests,
Nan

On Sun, Dec 26, 2010 at 8:25 PM, yipeng <[EMAIL PROTECTED]> wrote:

> Hi everyone,
>
> I would like to simulate network delay on 1 node in my cluster, perhaps by
> putting the thread to sleep every time it transfers data non-locally. I'm
> looking at the source but am not sure where to place the code. Is there a
> better way to do it... a tool perhaps? Or could someone point me in the
> right direction?
>
> Cheers,
>
> Yipeng
>

I'm trying to explore how Hadoop performs certain tasks (data deduplication
actually)  under such conditions.

Cheers,

Yipeng
On Sun, Dec 26, 2010 at 8:31 PM, Nan Zhu <[EMAIL PROTECTED]> wrote:

> Why would you like to *simulate* network delay? I haven't got your point,
>
> Bests,
> Nan
>
>
>
> On Sun, Dec 26, 2010 at 8:25 PM, yipeng <[EMAIL PROTECTED]> wrote:
>
> > Hi everyone,
> >
> > I would like to simulate network delay on 1 node in my cluster, perhaps
> by
> > putting the thread to sleep every time it transfers data non-locally. I'm
> > looking at the source but am not sure where to place the code. Is there a
> > better way to do it... a tool perhaps? Or could someone point me in the
> > right direction?
> >
> > Cheers,
> >
> > Yipeng
> >
>

If you're studying how the hadoop make the copies of the blocks, I'm not
familiar with HDFS, so I have no idea about it

If you're studying the data transfer between the Map and Reduce, I think
org.apache.hadoop.mapred.TaskTracker.MapOutputServlet is helpful

Bests
Nan

On Sun, Dec 26, 2010 at 8:56 PM, yipeng <[EMAIL PROTECTED]> wrote:

> I'm trying to explore how Hadoop performs certain tasks (data deduplication
> actually)  under such conditions.
>
> Cheers,
>
> Yipeng
>
>
> On Sun, Dec 26, 2010 at 8:31 PM, Nan Zhu <[EMAIL PROTECTED]> wrote:
>
> > Why would you like to *simulate* network delay? I haven't got your point,
> >
> > Bests,
> > Nan
> >
> >
> >
> > On Sun, Dec 26, 2010 at 8:25 PM, yipeng <[EMAIL PROTECTED]> wrote:
> >
> > > Hi everyone,
> > >
> > > I would like to simulate network delay on 1 node in my cluster, perhaps
> > by
> > > putting the thread to sleep every time it transfers data non-locally.
> I'm
> > > looking at the source but am not sure where to place the code. Is there
> a
> > > better way to do it... a tool perhaps? Or could someone point me in the
> > > right direction?
> > >
> > > Cheers,
> > >
> > > Yipeng
> > >
> >
>

Maybe this will help on Linux -
http://daniel.haxx.se/blog/2010/12/14/add-latency-to-localhost/
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-simulate-network-delay-on-1-node-tp2147265p2148277.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

See also https://github.com/toddlipcon/gremlins

On Sun, Dec 26, 2010 at 11:26 AM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:

> Hi there.
>
> What are looking at is fault injection.
> I am not sure what version of Hadoop you're looking at, but here's at
> what you take a look in 0.21 and forward:
>  - Herriot system testing framework (which does code instrumentation
> to add special APIs) on a real clusters. Here's some starting
> pointers:
>    - source code is in src/test/system
>    - http://wiki.apache.org/hadoop/HowToUseSystemTestFramework
>  - fault injection framework (should've been ported to 0.20 as well)
>    - Source code is under src/test/aop
>    - http://hadoop.apache.org/hdfs/docs/r0.21.0/faultinject_framework.html
>
> If you are running on simulated infrastructure you don't need to look
> further than fault injection framework. There's a test in HDFS which
> does pretty much what you're looking for but for pipe-lines (look
> under src/test/aop/org/apache/hadoop/hdfs/*).
>
> If you are on a physical cluster then you need to use a combination of
> 1st and 2nd. The implementation of faults in system tests are coming
> into Hadoop at some point of not very distant future, so you might
> want to wait a little bit.
> --
>   Take care,
> Konstantin (Cos) Boudnik
>
> On Sun, Dec 26, 2010 at 04:25, yipeng <[EMAIL PROTECTED]> wrote:
> > Hi everyone,
> >
> > I would like to simulate network delay on 1 node in my cluster, perhaps
> by
> > putting the thread to sleep every time it transfers data non-locally. I'm
> > looking at the source but am not sure where to place the code. Is there a
> > better way to do it... a tool perhaps? Or could someone point me in the
> > right direction?
> >
> > Cheers,
> >
> > Yipeng
> >
>

Hi guys, this is all very helpful.

Appreciate it. I will look into them.

Cheers,

Yipeng

On Mon, Dec 27, 2010 at 1:04 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> See also https://github.com/toddlipcon/gremlins
>
>
>
> On Sun, Dec 26, 2010 at 11:26 AM, Konstantin Boudnik <[EMAIL PROTECTED]>
> wrote:
>
> > Hi there.
> >
> > What are looking at is fault injection.
> > I am not sure what version of Hadoop you're looking at, but here's at
> > what you take a look in 0.21 and forward:
> >  - Herriot system testing framework (which does code instrumentation
> > to add special APIs) on a real clusters. Here's some starting
> > pointers:
> >    - source code is in src/test/system
> >    - http://wiki.apache.org/hadoop/HowToUseSystemTestFramework
> >  - fault injection framework (should've been ported to 0.20 as well)
> >    - Source code is under src/test/aop
> >    -
> http://hadoop.apache.org/hdfs/docs/r0.21.0/faultinject_framework.html
> >
> > If you are running on simulated infrastructure you don't need to look
> > further than fault injection framework. There's a test in HDFS which
> > does pretty much what you're looking for but for pipe-lines (look
> > under src/test/aop/org/apache/hadoop/hdfs/*).
> >
> > If you are on a physical cluster then you need to use a combination of
> > 1st and 2nd. The implementation of faults in system tests are coming
> > into Hadoop at some point of not very distant future, so you might
> > want to wait a little bit.
> > --
> >   Take care,
> > Konstantin (Cos) Boudnik
> >
> > On Sun, Dec 26, 2010 at 04:25, yipeng <[EMAIL PROTECTED]> wrote:
> > > Hi everyone,
> > >
> > > I would like to simulate network delay on 1 node in my cluster, perhaps
> > by
> > > putting the thread to sleep every time it transfers data non-locally.
> I'm
> > > looking at the source but am not sure where to place the code. Is there
> a
> > > better way to do it... a tool perhaps? Or could someone point me in the
> > > right direction?
> > >
> > > Cheers,
> > >
> > > Yipeng
> > >
> >
>