Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Uploading file to HDFS


Copy link to this message
-
Re: Uploading file to HDFS
You should be able to do it using WebHDFS.

WebHDFS is a REST API so there is no need to have hadoop installed on the
client where the file is located. YOu can find an example on how to copy a
file at the following URL.
http://hadoop.apache.org/docs/r1.0.4/webhdfs.html#CREATE

Olivier
On 19 April 2013 11:01, Wellington Chevreuil <[EMAIL PROTECTED]
> wrote:

> Can't you use flume for that?
>
>
> 2013/4/19 David Parks <[EMAIL PROTECTED]>
>
>> I just realized another trick you might trying. The Hadoop dfs client can
>> read input from STDIN, you could use netcat to pipe the stuff across to
>> HDFS without hitting the hard drive, I haven’t tried it, but here’s what I
>> would think might work:****
>>
>> ** **
>>
>> On the Hadoop box, open a listening port and feed that to the HDFS
>> command:****
>>
>> nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt****
>>
>> ** **
>>
>> On the remote server:****
>>
>> cat my_big_2tb_file > nc 10.1.1.1 2342****
>>
>> ** **
>>
>> I haven’t tried it yet, but in theory this would work. I just happened to
>> test out the hdfs dfs command reading from stdin. You might have to correct
>> the above syntax, I just wrote it off the top of my head.****
>>
>> ** **
>>
>> Dave****
>>
>> ** **
>>
>> ** **
>>
>> *From:* 超级塞亚人 [mailto:[EMAIL PROTECTED]]
>> *Sent:* Friday, April 19, 2013 11:35 AM
>> *To:* [EMAIL PROTECTED]
>> *Subject:* Uploading file to HDFS****
>>
>> ** **
>>
>> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
>> upload 2TB file to HDFS.How can I put the file to the namenode and upload
>> to HDFS? ****
>>
>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
[EMAIL PROTECTED]
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB