Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> Data Locality with WebHDFS


+
RJ Nowling 2014-03-17, 17:30
Copy link to this message
-
Re: Data Locality with WebHDFS
  1. All the specifics of Hadoop's operations are hidden in the source.
   That's a get-out clause of OSS, I know, but sometimes it's the clearest.
   2. For webhdfs I suspect it picks a local node with the data -you'd have
   to experiment to make sure
   3.  If webhdfs is missing fetaures, I'm sure they'd be welcome
   4. Hadoop 2.2+ uses protobuf for stable and cross platform IPC. , the
   listLocatedStatus() call on a filesystem will give you all the locations of
   blocks. Using that is another option -and probably higher performance- but
   is going to require more upfront engineering than GET calls. Sticking to
   WebHDFS -and extending it- is probably simpler

On 17 March 2014 17:29, RJ Nowling <[EMAIL PROTECTED]> wrote:
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.

 
+
RJ Nowling 2014-03-19, 19:40
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB