Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Automatically upload files into HDFS


Copy link to this message
-
Re: Automatically upload files into HDFS
kashif khan 2012-11-27, 21:25
Dear shekhar,

I am still struggling. I have written some java code, it does work but not
100%  as some time sending file with 0KB data. If you have any solution,
please let me know. I shall be very grateful to you.

Many thanks

Best regards

On Mon, Nov 26, 2012 at 4:42 PM, shekhar sharma <[EMAIL PROTECTED]>wrote:

> Hello Khasif,
> Sorry for late reply...Are you done? or u still struggling?
>
> mail me: [EMAIL PROTECTED]
> Regards,
> Som Shekhar Sharma
>
>
>
> On Wed, Nov 21, 2012 at 6:06 PM, kashif khan <[EMAIL PROTECTED]>wrote:
>
>> Dear Shankar Sharma.
>>
>> I am using Eclipse as IDE. I dont have any idea, how to create the
>> project as maven project. I have downloaded Mave2 but given me some strange
>> error. So if you can help me then I will try the maven. Actually, I am
>> trying to automatically upload the files into HDFS and then will apply some
>> algorithms to analyze the data. The algorithms will implement in mapreduce
>> . So if you think maven will good for me then please let me know how I can
>> create the project as maven project.
>>
>>
>> Many thanks
>>
>> Best regards,
>>
>> KK
>>
>>
>>
>>  On Tue, Nov 20, 2012 at 7:06 PM, shekhar sharma <[EMAIL PROTECTED]>wrote:
>>
>>> By the way how are you building and running your project.Are u running
>>> from any IDE?
>>> The best practises you can follow:
>>>
>>> (1) Create your project as maven project and give the dependency of
>>> hadoop-X.Y.Z . So your project will automatically will have all the
>>> necessary jars
>>> and i am sure you will not face these kind of errors
>>> (2) And In your HADOOP_CLASSPATH provide the path for $HADOOP_LIB.
>>>
>>> Regards,
>>> SOm
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Nov 20, 2012 at 9:52 PM, kashif khan <[EMAIL PROTECTED]>wrote:
>>>
>>>> Dear Tariq
>>>>
>>>> Many thanks, finally I have created the directory and upload the file.
>>>>
>>>> Once again many thanks
>>>>
>>>> Best regards
>>>>
>>>>
>>>> On Tue, Nov 20, 2012 at 3:04 PM, kashif khan <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Dear Many thanks
>>>>>
>>>>>
>>>>> I have downloaded the jar file and added to project. Now getting
>>>>> another error as:
>>>>>
>>>>> og4j:WARN No appenders could be found for logger
>>>>> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
>>>>> log4j:WARN Please initialize the log4j system properly.
>>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfigfor more info.
>>>>> Exception in thread "main" java.io.IOException: No FileSystem for
>>>>> scheme: hdfs
>>>>>     at
>>>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2206)
>>>>>     at
>>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2213)
>>>>>     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
>>>>>     at
>>>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)
>>>>>     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)
>>>>>
>>>>>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
>>>>>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:156)
>>>>>     at CopyFile.main(CopyFile.java:14)
>>>>>
>>>>> Have any idea about this?
>>>>>
>>>>> Thanks again
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Nov 20, 2012 at 2:53 PM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> You can download the jar here :
>>>>>> http://search.maven.org/remotecontent?filepath=com/google/guava/guava/13.0.1/guava-13.0.1.jar
>>>>>>
>>>>>> Regards,
>>>>>>     Mohammad Tariq
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 20, 2012 at 8:06 PM, kashif khan <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> Could please let me know the name of jar file and location
>>>>>>>
>>>>>>> Many thanks
>>>>>>>
>>>>>>> Best regards
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Nov 20, 2012 at 2:33 PM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>>>>>>>
>>>>>>>> Download the required jar and include it in your project.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>     Mohammad Tariq
>>>>>>>>
>>>>>>>>
>>>>>>>>