Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Direct HDFS access from a streaming job


Copy link to this message
-
Re: Direct HDFS access from a streaming job
On Mar 24, 2011, at 8:31 AM, Harsh J wrote:

> Hello,
>
> On Thu, Mar 24, 2011 at 8:45 PM, Keith Wiley <[EMAIL PROTECTED]> wrote:
>> Thanks.  Actually, I think that with reference to the passage I quoted in my first post, the unstated intent was to simply do a system() call and invoke "hadoop fs -get" or "hadoop fs -copyToLocal".
>
> Some would consider that a bit ugly in C/C++, but its almost the same
> thing I guess. For shell programs passed to streaming though, its a
> good solution.
I agree.  I like the API you found.  I was just saying that after posting my question, all references I found online that seemed to address that passage in the docs described the solution I then posted, that of directly invoking "hadoop get".

________________________________________________________________________________
Keith Wiley     [EMAIL PROTECTED]     keithwiley.com    music.keithwiley.com

"I do not feel obliged to believe that the same God who has endowed us with
sense, reason, and intellect has intended us to forgo their use."
                                           --  Galileo Galilei
________________________________________________________________________________
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB