Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Automatically upload files into HDFS


Copy link to this message
-
Re: Automatically upload files into HDFS
Mohammad Tariq 2012-11-20, 14:33
Download the required jar and include it in your project.

Regards,
    Mohammad Tariq

On Tue, Nov 20, 2012 at 7:57 PM, kashif khan <[EMAIL PROTECTED]> wrote:

> Dear Tariq Thanks
>
> I have added the jar files from Cdh and download the cdh4 eclipse plugin
> and copied into eclipse plugin folder. The previous error I think sorted
> out but now I am getting another strange error.
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/common/collect/Maps
>     at
> org.apache.hadoop.metrics2.lib.MetricsRegistry.<init>(MetricsRegistry.java:42)
>     at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.<init>(MetricsSystemImpl.java:87)
>     at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.<init>(MetricsSystemImpl.java:133)
>     at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:38)
>     at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:36)
>     at
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:97)
>     at
> org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:190)
>     at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2373)
>     at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2365)
>     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2233)
>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:156)
>     at CopyFile.main(CopyFile.java:14)
> Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Maps
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>     ... 13 more
>
> Have any idea about this error.
>
> Many thanks
>
>
>
>
>
> On Tue, Nov 20, 2012 at 2:19 PM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>
>> Hello Kashif,
>>
>>      You are correct. This because of some version mismatch. I am not
>> using CDH personally but AFAIK, CDH4 uses Hadoop-2.x.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>>
>> On Tue, Nov 20, 2012 at 4:10 PM, kashif khan <[EMAIL PROTECTED]>wrote:
>>
>>> HI M Tariq
>>>
>>>
>>> I am trying the following the program to create directory and copy file
>>> to hdfs. But I am getting the following errors
>>>
>>>
>>>
>>> Program:
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.FileSystem;
>>> import org.apache.hadoop.fs.Path;
>>> import java.io.IOException;
>>>
>>> public class CopyFile {
>>>
>>>
>>>         public static void main(String[] args) throws IOException{
>>>         Configuration conf = new Configuration();
>>>          conf.set("fs.default.name", "hadoop1.example.com:8020");
>>>         FileSystem dfs = FileSystem.get(conf);
>>>         String dirName = "Test1";
>>>         Path src = new Path(dfs.getWorkingDirectory() + "/" + dirName);
>>>         dfs.mkdirs(src);
>>>         Path scr1 = new Path("/usr/Eclipse/Output.csv");
>>>         Path dst = new Path(dfs.getWorkingDirectory() + "/Test1/");
>>>         dfs.copyFromLocalFile(src, dst);
>>>
>>>         }
>>>         }
>>>
>>>
>>>     Exception in thread "main" org.apache.hadoop.ipc.RemoteException:
>>> Server IPC version 7 cannot communicate with client version 4
>>>     at org.apache.hadoop.ipc.Client.call(Client.java:1070)
>>>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>>>     at $Proxy1.getProtocolVersion(Unknown Source)
>>>     at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
>>>     at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
>>>     at
>>> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)