Hi,

On Tue, Sep 6, 2011 at 9:29 AM, Ralf Heyde <[EMAIL PROTECTED]> wrote:
> Hello,
>
>
>
> I have found a HDFSClient which shows me, how to access my HDFS from inside
> the cluster (i.e. running on a Node).
>
>
>
> My Idea is, that different processes may write 64M Chunks to HDFS from
> external Sources/Clients.
>
> Is that possible?

Yes, the same HDFSClient code you have above should work outside the
cluster, you just need core-site.xml and hdfs-site.xml in your
classpath so client knows where the namenode is and what the block
size should be.

Brock

Hi,

 It is very much possible. Infact that is the main use case for Hadoop :-)

 You need to put the hadoop-hdfs*.jar hdoop-common*.jar's in your class path from where you want to run the client program.

At client node side use the below sample code
 
 Configuration conf=new Configuration(); //you can set the required  configurations here
 FileSystem fs =new DistributedFileSystem();
 fs.initialize(new URI(<Name_Node_URL>), conf);

 fs.copyToLocal(srcPath, destPath)
 fs.copyFromLocal(srcPath,destPath)
 .....etc
 There are many API exposed in FileSystem.java class. So, you can make use of them.
Regards,
Uma
 

----- Original Message -----
From: Ralf Heyde <[EMAIL PROTECTED]>
Date: Monday, September 5, 2011 7:59 pm
Subject: Is it possible to access the HDFS via Java OUTSIDE the Cluster?
To: [EMAIL PROTECTED]

> Hello,
>
>
>
> I have found a HDFSClient which shows me, how to access my HDFS
> from inside
> the cluster (i.e. running on a Node).
>
>
>
> My Idea is, that different processes may write 64M Chunks to HDFS from
> external Sources/Clients.
>
> Is that possible?
>
> How that can be done? Does anybody have some Example Code?
>
>
>
> Thanks,
>
>
>
> Ralf
>
>
>
>

Good evening,

this topic seems very interesting.
To be sure I understood the case - do you mean that I can write a simple Java program and access a file stored in HDFS from within the java application?

Assuming that I have e.g. 10 files of size 30GB each stored on HDFS on a cluster of 15 nodes, how can I run a java program that accesses these files and reads some blocks from them? Is it possible to do it without copying the files via -copyToLocal ?

If yes, could anyone give some general directions on the general form of such a java code, and on how to run such a program?

Thank  you in advance
Sofia

________________________________
From: Uma Maheswara Rao G 72686 <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, September 5, 2011 6:04 PM
Subject: Re: Is it possible to access the HDFS via Java OUTSIDE the Cluster?

Hi,

It is very much possible. Infact that is the main use case for Hadoop :-)

You need to put the hadoop-hdfs*.jar hdoop-common*.jar's in your class path from where you want to run the client program.

At client node side use the below sample code

Configuration conf=new Configuration(); //you can set the required  configurations here
FileSystem fs =new DistributedFileSystem();
fs.initialize(new URI(<Name_Node_URL>), conf);

fs.copyToLocal(srcPath, destPath)
fs.copyFromLocal(srcPath,destPath)
.....etc
There are many API exposed in FileSystem.java class. So, you can make use of them.
Regards,
Uma
----- Original Message -----
From: Ralf Heyde <[EMAIL PROTECTED]>
Date: Monday, September 5, 2011 7:59 pm
Subject: Is it possible to access the HDFS via Java OUTSIDE the Cluster?
To: [EMAIL PROTECTED]

> Hello,
>
>
>
> I have found a HDFSClient which shows me, how to access my HDFS
> from inside
> the cluster (i.e. running on a Node).
>
>
>
> My Idea is, that different processes may write 64M Chunks to HDFS from
> external Sources/Clients.
>
> Is that possible?
>
> How that can be done? Does anybody have some Example Code?
>
>
>
> Thanks,
>
>
>
> Ralf
>
>
>
>

Hi,

I think it goes even beyond that. By using FUSE and sshFS you can also
mount the hdfs folder in your local directory and treat it as any other
local folder.

BR,
George

On 9/5/2011 6:42 PM, Sofia Georgiakaki wrote:
> Good evening,
>
> this topic seems very interesting.
> To be sure I understood the case - do you mean that I can write a simple Java program and access a file stored in HDFS from within the java application?
>
> Assuming that I have e.g. 10 files of size 30GB each stored on HDFS on a cluster of 15 nodes, how can I run a java program that accesses these files and reads some blocks from them? Is it possible to do it without copying the files via -copyToLocal ?
>
> If yes, could anyone give some general directions on the general form of such a java code, and on how to run such a program?
>
> Thank  you in advance
> Sofia
>
>
>
>
>
> ________________________________
> From: Uma Maheswara Rao G 72686<[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Monday, September 5, 2011 6:04 PM
> Subject: Re: Is it possible to access the HDFS via Java OUTSIDE the Cluster?
>
> Hi,
>
> It is very much possible. Infact that is the main use case for Hadoop :-)
>
> You need to put the hadoop-hdfs*.jar hdoop-common*.jar's in your class path from where you want to run the client program.
>
> At client node side use the below sample code
>
> Configuration conf=new Configuration(); //you can set the required  configurations here
> FileSystem fs =new DistributedFileSystem();
> fs.initialize(new URI(<Name_Node_URL>), conf);
>
> fs.copyToLocal(srcPath, destPath)
> fs.copyFromLocal(srcPath,destPath)
> .....etc
> There are many API exposed in FileSystem.java class. So, you can make use of them.
>
>
> Regards,
> Uma
>
>
> ----- Original Message -----
> From: Ralf Heyde<[EMAIL PROTECTED]>
> Date: Monday, September 5, 2011 7:59 pm
> Subject: Is it possible to access the HDFS via Java OUTSIDE the Cluster?
> To: [EMAIL PROTECTED]
>
>> Hello,
>>
>>
>>
>> I have found a HDFSClient which shows me, how to access my HDFS
>> from inside
>> the cluster (i.e. running on a Node).
>>
>>
>>
>> My Idea is, that different processes may write 64M Chunks to HDFS from
>> external Sources/Clients.
>>
>> Is that possible?
>>
>> How that can be done? Does anybody have some Example Code?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Ralf
>>
>>
>>
>>
--

---------------------------

George Kousiouris
Electrical and Computer Engineer
Division of Communications,
Electronics and Information Engineering
School of Electrical and Computer Engineering
Tel: +30 210 772 2546
Mobile: +30 6939354121
Fax: +30 210 772 2569
Email: [EMAIL PROTECTED]
Site: http://users.ntua.gr/gkousiou/

National Technical University of Athens
9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

On 09/05/2011 05:42 PM, Sofia Georgiakaki wrote:
> Good evening,
>
> this topic seems very interesting. To be sure I understood the case -
> do you mean that I can write a simple Java program and access a file
> stored in HDFS from within the java application?
>
> Assuming that I have e.g. 10 files of size 30GB each stored on HDFS
> on a cluster of 15 nodes, how can I run a java program that accesses
> these files and reads some blocks from them? Is it possible to do it
> without copying the files via -copyToLocal ?
>
> If yes, could anyone give some general directions on the general form
> of such a java code, and on how to run such a program?
>
> Thank  you in advance Sofia

You certainly can access a file on HDFS through a simple Java program.
You can also access your files with an even simpler Python program using
the Pydoop HDFS module (http://pydoop.sf.net/).  Here's a simple Python
script to print a file:
import pydoop.hdfs as py_hdfs

fs = py_hdfs.hdfs('default', 0)

for line in fs.open_file("myfile", 'r'):
print line

--
Luca Pireddu
CRS4 - Distributed Computing Group
Loc. Pixina Manna Edificio 1
Pula 09010 (CA), Italy
Tel: +39 0709250452

Use the Thrift HDFS API.
On Tue, 6 Sep 2011 19:28:07 +0200, "Ralf Heyde" <[EMAIL PROTECTED]> wrote:
> Yeah it works. ...
> I just copied the core-site.xml and hdfs-site.xml ... This setup does
not
> work.
> After copying the whole hadoop-installation-folder from the master-node
...
> it works.
>
> Thanks.
>
> -----Original Message-----
> From: Uma Maheswara Rao G 72686 [mailto:[EMAIL PROTECTED]]
> Sent: Montag, 5. September 2011 17:04
> To: [EMAIL PROTECTED]
> Subject: Re: Is it possible to access the HDFS via Java OUTSIDE the
> Cluster?
>
> Hi,
>
>  It is very much possible. Infact that is the main use case for Hadoop
:-)
>
>  You need to put the hadoop-hdfs*.jar hdoop-common*.jar's in your class
>  path
> from where you want to run the client program.
>
> At client node side use the below sample code
>  
>  Configuration conf=new Configuration(); //you can set the required
> configurations here
>  FileSystem fs =new DistributedFileSystem();
>  fs.initialize(new URI(<Name_Node_URL>), conf);
>
>  fs.copyToLocal(srcPath, destPath)
>  fs.copyFromLocal(srcPath,destPath)
>  .....etc
>  There are many API exposed in FileSystem.java class. So, you can make
use
> of them.
>
>
> Regards,
> Uma
>  
>
> ----- Original Message -----
> From: Ralf Heyde <[EMAIL PROTECTED]>
> Date: Monday, September 5, 2011 7:59 pm
> Subject: Is it possible to access the HDFS via Java OUTSIDE the Cluster?
> To: [EMAIL PROTECTED]
>
>> Hello,
>>
>>
>>
>> I have found a HDFSClient which shows me, how to access my HDFS
>> from inside
>> the cluster (i.e. running on a Node).
>>
>>
>>
>> My Idea is, that different processes may write 64M Chunks to HDFS from
>> external Sources/Clients.
>>
>> Is that possible?
>>
>> How that can be done? Does anybody have some Example Code?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Ralf
>>
>>
>>
>>

Hello,

 

I have found a HDFSClient which shows me, how to access my HDFS from inside
the cluster (i.e. running on a Node).

 

My Idea is, that different processes may write 64M Chunks to HDFS from
external Sources/Clients.

Is that possible?

How that can be done? Does anybody have some Example Code?

 

Thanks,

 

Ralf

 

Yeah it works. ...
I just copied the core-site.xml and hdfs-site.xml ... This setup does not
work.
After copying the whole hadoop-installation-folder from the master-node ...
it works.

Thanks.

-----Original Message-----
From: Uma Maheswara Rao G 72686 [mailto:[EMAIL PROTECTED]]
Sent: Montag, 5. September 2011 17:04
To: [EMAIL PROTECTED]
Subject: Re: Is it possible to access the HDFS via Java OUTSIDE the Cluster?

Hi,

 It is very much possible. Infact that is the main use case for Hadoop :-)

 You need to put the hadoop-hdfs*.jar hdoop-common*.jar's in your class path
from where you want to run the client program.

At client node side use the below sample code
 
 Configuration conf=new Configuration(); //you can set the required
configurations here
 FileSystem fs =new DistributedFileSystem();
 fs.initialize(new URI(<Name_Node_URL>), conf);

 fs.copyToLocal(srcPath, destPath)
 fs.copyFromLocal(srcPath,destPath)
 .....etc
 There are many API exposed in FileSystem.java class. So, you can make use
of them.
Regards,
Uma
 

----- Original Message -----
From: Ralf Heyde <[EMAIL PROTECTED]>
Date: Monday, September 5, 2011 7:59 pm
Subject: Is it possible to access the HDFS via Java OUTSIDE the Cluster?
To: [EMAIL PROTECTED]

> Hello,
>
>
>
> I have found a HDFSClient which shows me, how to access my HDFS
> from inside
> the cluster (i.e. running on a Node).
>
>
>
> My Idea is, that different processes may write 64M Chunks to HDFS from
> external Sources/Clients.
>
> Is that possible?
>
> How that can be done? Does anybody have some Example Code?
>
>
>
> Thanks,
>
>
>
> Ralf
>
>
>
>