Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Re: How to copy log files from remote windows machine to Hadoop cluster


Copy link to this message
-
Re: How to copy log files from remote windows machine to Hadoop cluster
Mahesh Balija 2013-01-17, 11:17
That link talks about just installing Flume on Windows machine (NOT even
have configs to push logs to the Hadoop cluster), but what if I have to
collect logs from various clients, then I will endup installing in all
clients.

I have installed Flume successfully on Linux but I have to configure it
such a way that it should gather the log files from the remote windows box?

Harsh can you throw some light on this?

On Thu, Jan 17, 2013 at 4:21 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Yes. It is possible. I haven't tries windows+flume+hadoop combo
> personally, but it should work. You may find this link<http://mapredit.blogspot.in/2012/07/run-flume-13x-on-windows.html>useful. Alex
> has explained beautifully how to run Flume on a windows box.If I
> get time i'll try to simulate your use case and let you know.
>
> BTW, could you please share with us whatever you have tried??
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Jan 17, 2013 at 4:09 PM, Mahesh Balija <[EMAIL PROTECTED]
> > wrote:
>
>> I have studied Flume but I didn't find any thing useful in my case.
>> My requirement is there is a directory in Windows machine, in which the
>> files will be generated and keep updated with new logs. I want to have a
>> tail kind of mechanism (using exec source) through which I can push the
>> latest updates into the cluster.
>> Or I have to simply push once in a day to the cluster using spooling
>> directory mechanism.
>>
>> Can somebody assist whether it is possible using Flume if so the
>> configurations needed for this specific to remote windows machine.
>>
>> But
>>
>> On Thu, Jan 17, 2013 at 3:48 PM, Mirko Kämpf <[EMAIL PROTECTED]>wrote:
>>
>>> Give Flume (http://flume.apache.org/) a chance to collect your data.
>>>
>>> Mirko
>>>
>>>
>>>
>>> 2013/1/17 sirenfei <[EMAIL PROTECTED]>
>>>
>>>> ftp auto upload?
>>>>
>>>>
>>>> 2013/1/17 Mahesh Balija <[EMAIL PROTECTED]>:
>>>> > the Hadoop cluster (HDFS) either in synchronous or asynchronou
>>>>
>>>
>>>
>>
>