Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # dev - Re: SDP support for Hadoop RPC


+
Milind Bhandarkar 2013-10-09, 00:57
+
Yasin Celik 2013-10-09, 01:13
+
Milind Bhandarkar 2013-10-09, 01:19
+
Steve Loughran 2013-10-09, 08:45
+
Milind Bhandarkar 2013-10-09, 09:09
+
Suresh Srinivas 2013-10-09, 15:47
Copy link to this message
-
Re: SDP support for Hadoop RPC
Milind Bhandarkar 2013-10-09, 17:38
I cannot believe i made the same mistake twice, of reply-to to mailing list. I need to get off the outlook client.

once again, sorry folks. this was supposed to go to steve's personal email.

- milind

Sent from my iPhone

> On Oct 9, 2013, at 8:47, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
>
> Milind, please stop this. The topic here is not what Steve's employers
> wants to sell or recommend. Please stick to the technical issue. This is
> the second time this week where a thread unnecessarily goes beyond
> technical issues. If you have an axe to grind, please keep it off this
> forum. It is getting annoying.
>
> I am also annoyed by the frequent references to private communications made
> in this public forum.
>
>> Let's meet in the same chinese restaurant, and I will give you the
> numbers on performance and cost
> Instead of conspiracy theories and employer bashing, adding details such as
> this is what is useful to this forum!
>
>
> On Wed, Oct 9, 2013 at 2:09 AM, Milind Bhandarkar <[EMAIL PROTECTED]
>> wrote:
>
>> No, Steve, I meant exactly what I wrote. One day, when you are here, Let's
>> meet in the same chinese restaurant, and I will give you the numbers on
>> performance and cost, and let us do the division of these two numbers.
>>
>> Let's talk about how the nn latency becomes a bottleneck for rest of the
>> cluster's throughput, and why the networking world's advances cannot be
>> pushed under the rug.
>>
>> Let's talk about why your employers are cozying up with DSSD and engenio
>> while you and others in open source are insisting on 1GbE and DAS SATA
>> disks being the most suitable  for Hadoop.
>>
>> And most of all, lets chat about why business aspects of Hadoop are acting
>> against the open source from the same orgs' folks.
>>
>> Tomorrow and day after we are conducting big data benchmarking workshop in
>> San Jose, where your partners and other open-core hadoop company's partners
>> will demonstrate how advanced hardware (cpus, networks, storage) is more
>> cost effective that what you are recommending.
>>
>> I had recognized this phenomenon very early, and wrote a blog post
>> comparing open source hadoop development to charlie chaplin, who missed the
>> color and talky movie technology, by sticking to mute black and white
>> technology. I know your employer has moved beyond cheap hardware, based on
>> what I hear from customers where we compete. I an wondering why you still
>> keep on insisting new technologies are not worth it.
>>
>> - milind
>>
>> Sent from my iPhone
>>
>>> On Oct 9, 2013, at 1:45, Steve Loughran <[EMAIL PROTECTED]> wrote:
>>>
>>> On 9 October 2013 01:57, Milind Bhandarkar <[EMAIL PROTECTED]
>>> wrote:
>>>
>>>> Yes, we have. It works very well, but it is considered too niche by
>> folks
>>>> who insist on buying the least capable hardware for their test clusters,
>>>> and therefore, recommend such underpowered clusters to customers as
>> well.
>>>
>>>
>>> surely you meant to say  "take advantage of the cost model of JBOD
>> storage
>>> and ethernet to allow data to be stored and accessed at significantly
>> lower
>>> price points than for legacy storage architectures and pricing models -so
>>> enabling their customers to store and process data they would have
>>> previously had to discard" (0)
>>>
>>> IB should be most interesting at the app level -for apps > classic MR.
>>> That's giraph, streaming work, Tez. I'd like to see some numbers there.
>> As
>>> the oracle
>>>
>>> For storage, IB would make locality less of an issue (1,2), and instead
>>> make the level of storage: SSD vs HDD more significant in terms of
>>> performance (2). There is ongoing work there  in a set of JIRAs about
>>> multi-tier storage.
>>>
>>> I don't know the current state of Hadoop on IB, or even if
>> allocateDirect()
>>> of NIO has been picked up. For IPC there should be some latency
>>> improvements, while for the Datanodes its the bulk data you want to push
>
+
Steve Loughran 2013-10-09, 08:50
+
Luke Lu 2013-10-10, 08:00
+
Steve Loughran 2013-10-10, 08:41
+
Yasin Celik 2013-10-10, 12:12
+
Luke Lu 2013-10-10, 19:12