Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Automatically upload files into HDFS


Copy link to this message
-
Re: Automatically upload files into HDFS
kashif khan 2012-11-20, 10:40
HI M Tariq
I am trying the following the program to create directory and copy file to
hdfs. But I am getting the following errors

Program:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.IOException;

public class CopyFile {

        public static void main(String[] args) throws IOException{
        Configuration conf = new Configuration();
        conf.set("fs.default.name", "hadoop1.example.com:8020");
        FileSystem dfs = FileSystem.get(conf);
        String dirName = "Test1";
        Path src = new Path(dfs.getWorkingDirectory() + "/" + dirName);
        dfs.mkdirs(src);
        Path scr1 = new Path("/usr/Eclipse/Output.csv");
        Path dst = new Path(dfs.getWorkingDirectory() + "/Test1/");
        dfs.copyFromLocalFile(src, dst);

        }
        }
    Exception in thread "main" org.apache.hadoop.ipc.RemoteException:
Server IPC version 7 cannot communicate with client version 4
    at org.apache.hadoop.ipc.Client.call(Client.java:1070)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at $Proxy1.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
    at CopyFile.main(CopyFile.java:11)

I am using CDH4.1. i have download the source file of hadoop-1.0.4 and
import the jar files into Eclipse. I think it is due to version problem.
Could you please let me know what will be correct version for the CDH4.1?

Many thanks

On Mon, Nov 19, 2012 at 3:41 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> It should work. Same code is working fine for me. Try to create some other
> directory in your Hdfs and use it as your output path. Also see if you find
> something in datanode logs.
>
> Regards,
>     Mohammad Tariq
>
>
>
> On Mon, Nov 19, 2012 at 9:04 PM, kashif khan <[EMAIL PROTECTED]>wrote:
>
>> The input path is fine. Problem in output path. I am just wonder that it
>> copy the data into local disk  (/user/root/) not into hdfs. I dont know
>> why? Is it we give the correct statement to point to hdfs?
>>
>> Thanks
>>
>>
>>
>> On Mon, Nov 19, 2012 at 3:10 PM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>>
>>> Try this as your input file path
>>> Path inputFile = new Path("file:///usr/Eclipse/Output.csv");
>>>
>>> Regards,
>>>     Mohammad Tariq
>>>
>>>
>>>
>>> On Mon, Nov 19, 2012 at 8:31 PM, kashif khan <[EMAIL PROTECTED]>wrote:
>>>
>>>> when I am applying the command as
>>>>
>>>> $ hadoop fs -put /usr/Eclipse/Output.csv /user/root/Output.csv.
>>>>
>>>> its work fine and file browsing in the hdfs. But i dont know why its
>>>> not work in program.
>>>>
>>>> Many thanks for your cooperation.
>>>>
>>>> Best regards,
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Nov 19, 2012 at 2:53 PM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> It would be good if I could have a look on the files. Meantime try
>>>>> some other directories. Also, check the directory permissions once.
>>>>>
>>>>> Regards,
>>>>>     Mohammad Tariq
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Nov 19, 2012 at 8:13 PM, kashif khan <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>>
>>>>>> I have tried through root user and made the following changes:
>>>>>>
>>>>>>
>>>>>> Path inputFile = new Path("/usr/Eclipse/Output.csv");
>>>>>> Path outputFile = new Path("/user/root/Output1.csv");