Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Flume to stream logs live


+
Kartashov, Andy 2012-12-14, 14:47
+
Mohammad Tariq 2012-12-14, 14:51
+
Brock Noland 2012-12-14, 14:52
+
Paul Chavez 2012-12-14, 16:24
+
Bertrand Dechoux 2012-12-16, 12:25
+
Juhani Connolly 2012-12-17, 01:33
+
Alexander Lorenz 2012-12-17, 07:57
+
Kartashov, Andy 2012-12-19, 16:20
+
Alexander Alten-Lorenz 2012-12-20, 07:02
Copy link to this message
-
RE: Flume to stream logs live
Thank you guys.

Managed to install flume-ng  and run flume agent on windows at CMD line using java VM, i.e C:> java -Dlog4j.configuration= .... -cp "...\lib\*" -f "flume.conf" -n agent-test

Having source - seq / and sink - logger appeared to be working.
Having source - syslogTcp / and sink - logger appeared to be working, or did it?? See below/ Every second lifestatusSupervisor would advise : Status ceck complete.  What's the next step.
Can you actually send data to a certain port. Can you try running the flume client or execute bash -tail command somehow in Windows without running cygwin?
[cid:[EMAIL PROTECTED]BEA0]
From: Juhani Connolly [mailto:[EMAIL PROTECTED]]
Sent: Sunday, December 16, 2012 8:33 PM
To: [EMAIL PROTECTED]
Subject: Re: Flume to stream logs live

If you can't get flume running on windows your alternatives look like syslog, scribed and fluentd. We used scribed before moving to flume, but tbh, it can be a pain to compile and the hdfs support on it is not the best, we had data loss if there was an hdfs restart etc. I can't say much about the other two

On 12/14/2012 11:47 PM, Kartashov, Andy wrote:
Flummers,

Loved working with Flume 1.2 - very easy and simple configuration, it was a pleasure to work with. Managed to "tail -F" logs from unix server and into a hdfs cluster. The problem started when I also needed to push logs from a Windows application server.  Spent three days researching on how to install flume on Windows and run  a deamon/agent that will push the logs to the Avro source I successfully configured and ran on Unix. No luck. So I am looking t alternative. Is there other framework available out there to help me with my issue. What about scribe?

Andy Kartashov
MPAC
IT Architecture, Co-op
1340 Pickering Parkway, Pickering, L1V 0C4
* Phone : (905) 837 6269
* Mobile: (416) 722 1787
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
+
Kartashov, Andy 2012-12-20, 17:44