Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> Re: SDP support for Hadoop RPC


+
Milind Bhandarkar 2013-10-09, 00:57
+
Yasin Celik 2013-10-09, 01:13
+
Milind Bhandarkar 2013-10-09, 01:19
+
Steve Loughran 2013-10-09, 08:45
+
Milind Bhandarkar 2013-10-09, 09:09
+
Suresh Srinivas 2013-10-09, 15:47
Copy link to this message
-
Re: SDP support for Hadoop RPC
I cannot believe i made the same mistake twice, of reply-to to mailing list. I need to get off the outlook client.

once again, sorry folks. this was supposed to go to steve's personal email.

- milind

Sent from my iPhone

> On Oct 9, 2013, at 8:47, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
>
> Milind, please stop this. The topic here is not what Steve's employers
> wants to sell or recommend. Please stick to the technical issue. This is
> the second time this week where a thread unnecessarily goes beyond
> technical issues. If you have an axe to grind, please keep it off this
> forum. It is getting annoying.
>
> I am also annoyed by the frequent references to private communications made
> in this public forum.
>
>> Let's meet in the same chinese restaurant, and I will give you the
> numbers on performance and cost
> Instead of conspiracy theories and employer bashing, adding details such as
> this is what is useful to this forum!
>
>
> On Wed, Oct 9, 2013 at 2:09 AM, Milind Bhandarkar <[EMAIL PROTECTED]
>> wrote:
>
>> No, Steve, I meant exactly what I wrote. One day, when you are here, Let's
>> meet in the same chinese restaurant, and I will give you the numbers on
>> performance and cost, and let us do the division of these two numbers.
>>
>> Let's talk about how the nn latency becomes a bottleneck for rest of the
>> cluster's throughput, and why the networking world's advances cannot be
>> pushed under the rug.
>>
>> Let's talk about why your employers are cozying up with DSSD and engenio
>> while you and others in open source are insisting on 1GbE and DAS SATA
>> disks being the most suitable  for Hadoop.
>>
>> And most of all, lets chat about why business aspects of Hadoop are acting
>> against the open source from the same orgs' folks.
>>
>> Tomorrow and day after we are conducting big data benchmarking workshop in
>> San Jose, where your partners and other open-core hadoop company's partners
>> will demonstrate how advanced hardware (cpus, networks, storage) is more
>> cost effective that what you are recommending.
>>
>> I had recognized this phenomenon very early, and wrote a blog post
>> comparing open source hadoop development to charlie chaplin, who missed the
>> color and talky movie technology, by sticking to mute black and white
>> technology. I know your employer has moved beyond cheap hardware, based on
>> what I hear from customers where we compete. I an wondering why you still
>> keep on insisting new technologies are not worth it.
>>
>> - milind
>>
>> Sent from my iPhone
>>
>>> On Oct 9, 2013, at 1:45, Steve Loughran <[EMAIL PROTECTED]> wrote:
>>>
>>> On 9 October 2013 01:57, Milind Bhandarkar <[EMAIL PROTECTED]
>>> wrote:
>>>
>>>> Yes, we have. It works very well, but it is considered too niche by
>> folks
>>>> who insist on buying the least capable hardware for their test clusters,
>>>> and therefore, recommend such underpowered clusters to customers as
>> well.
>>>
>>>
>>> surely you meant to say  "take advantage of the cost model of JBOD
>> storage
>>> and ethernet to allow data to be stored and accessed at significantly
>> lower
>>> price points than for legacy storage architectures and pricing models -so
>>> enabling their customers to store and process data they would have
>>> previously had to discard" (0)
>>>
>>> IB should be most interesting at the app level -for apps > classic MR.
>>> That's giraph, streaming work, Tez. I'd like to see some numbers there.
>> As
>>> the oracle
>>>
>>> For storage, IB would make locality less of an issue (1,2), and instead
>>> make the level of storage: SSD vs HDD more significant in terms of
>>> performance (2). There is ongoing work there  in a set of JIRAs about
>>> multi-tier storage.
>>>
>>> I don't know the current state of Hadoop on IB, or even if
>> allocateDirect()
>>> of NIO has been picked up. For IPC there should be some latency
>>> improvements, while for the Datanodes its the bulk data you want to push
>
+
Steve Loughran 2013-10-09, 08:50
+
Luke Lu 2013-10-10, 08:00
+
Steve Loughran 2013-10-10, 08:41
+
Yasin Celik 2013-10-10, 12:12
+
Luke Lu 2013-10-10, 19:12
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB