Flummers,

Loved working with Flume 1.2 - very easy and simple configuration, it was a pleasure to work with. Managed to "tail -F" logs from unix server and into a hdfs cluster. The problem started when I also needed to push logs from a Windows application server.  Spent three days researching on how to install flume on Windows and run  a deamon/agent that will push the logs to the Avro source I successfully configured and ran on Unix. No luck. So I am looking t alternative. Is there other framework available out there to help me with my issue. What about scribe?

Andy Kartashov
MPAC
IT Architecture, Co-op
1340 Pickering Parkway, Pickering, L1V 0C4
* Phone : (905) 837 6269
* Mobile: (416) 722 1787
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel

If you can't get flume running on windows your alternatives look like
syslog, scribed and fluentd. We used scribed before moving to flume, but
tbh, it can be a pain to compile and the hdfs support on it is not the
best, we had data loss if there was an hdfs restart etc. I can't say
much about the other two

On 12/14/2012 11:47 PM, Kartashov, Andy wrote:
>
> Flummers,
>
> Loved working with Flume 1.2 -- very easy and simple configuration, it
> was a pleasure to work with. Managed to "tail --F" logs from unix
> server and into a hdfs cluster. The problem started when I also needed
> to push logs from a Windows application server.  Spent three days
> researching on how to install flume on Windows and run  a deamon/agent
> that will push the logs to the Avro source I successfully configured
> and ran on Unix. No luck. So I am looking t alternative. Is there
> other framework available out there to help me with my issue. What
> about scribe?
>
> Andy Kartashov
>
> *MPAC*
>
> IT Architecture, Co-op
>
> 1340 Pickering Parkway, Pickering, L1V 0C4
>
> (Phone : (905) 837 6269
>
> (Mobile: (416) 722 1787
>
> *[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>*
>
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use,
> copying or disclosure is prohibited. If you are not the intended
> recipient, please delete and contact the sender immediately. Please
> consider the environment before printing this e-mail. AVIS : le
> pr�sent courriel et toute pi�ce jointe qui l'accompagne sont
> confidentiels, prot�g�s par le droit d'auteur et peuvent �tre couverts
> par le secret professionnel. Toute utilisation, copie ou divulgation
> non autoris�e est interdite. Si vous n'�tes pas le destinataire pr�vu
> de ce courriel, supprimez-le et contactez imm�diatement l'exp�diteur.
> Veuillez penser � l'environnement avant d'imprimer le pr�sent courriel

Hey,

fluentd is a good tip, they have a custom sink:
https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Plugins

This could work.

Best,
 Alex

On Dec 17, 2012, at 2:33 AM, Juhani Connolly <[EMAIL PROTECTED]> wrote:

> If you can't get flume running on windows your alternatives look like syslog, scribed and fluentd. We used scribed before moving to flume, but tbh, it can be a pain to compile and the hdfs support on it is not the best, we had data loss if there was an hdfs restart etc. I can't say much about the other two
>
> On 12/14/2012 11:47 PM, Kartashov, Andy wrote:
>>
>> Flummers,
>>
>> Loved working with Flume 1.2 -- very easy and simple configuration, it was a pleasure to work with. Managed to "tail --F" logs from unix server and into a hdfs cluster. The problem started when I also needed to push logs from a Windows application server.  Spent three days researching on how to install flume on Windows and run  a deamon/agent that will push the logs to the Avro source I successfully configured and ran on Unix. No luck. So I am looking t alternative. Is there other framework available out there to help me with my issue. What about scribe?
>>
>> Andy Kartashov
>>
>> *MPAC*
>>
>> IT Architecture, Co-op
>>
>> 1340 Pickering Parkway, Pickering, L1V 0C4
>>
>> (Phone : (905) 837 6269
>>
>> (Mobile: (416) 722 1787
>>
>> *[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>*
>>
>> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
>

Thank you for your suggestions: Mr. Chavez , Mr. Lorenz, Mr. Dechoux and Mr. Connolly,

I just managed to successfully configure flume to push logs from windows box, running behind Proxy,  to the Avro-agent, running on Linux box, and without Cygwin.. by just running JVM process. So happy.

Cheers,
AK47
-----Original Message-----
From: Alexander Lorenz [mailto:[EMAIL PROTECTED]]
Sent: Monday, December 17, 2012 2:58 AM
To: [EMAIL PROTECTED]
Subject: Re: Flume to stream logs live

Hey,

fluentd is a good tip, they have a custom sink:
https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Plugins

This could work.

Best,
 Alex

On Dec 17, 2012, at 2:33 AM, Juhani Connolly <[EMAIL PROTECTED]> wrote:

> If you can't get flume running on windows your alternatives look like syslog, scribed and fluentd. We used scribed before moving to flume, but tbh, it can be a pain to compile and the hdfs support on it is not the best, we had data loss if there was an hdfs restart etc. I can't say much about the other two
>
> On 12/14/2012 11:47 PM, Kartashov, Andy wrote:
>>
>> Flummers,
>>
>> Loved working with Flume 1.2 -- very easy and simple configuration, it was a pleasure to work with. Managed to "tail --F" logs from unix server and into a hdfs cluster. The problem started when I also needed to push logs from a Windows application server.  Spent three days researching on how to install flume on Windows and run  a deamon/agent that will push the logs to the Avro source I successfully configured and ran on Unix. No luck. So I am looking t alternative. Is there other framework available out there to help me with my issue. What about scribe?
>>

>>

>>
>> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
>

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

Hey AK47,

good to know, can you describe for the community what exactly you've done and what you've archived? Would be great,

cheers,
 Alex

On Dec 19, 2012, at 5:20 PM, "Kartashov, Andy" <[EMAIL PROTECTED]> wrote:

> Thank you for your suggestions: Mr. Chavez , Mr. Lorenz, Mr. Dechoux and Mr. Connolly,
>
> I just managed to successfully configure flume to push logs from windows box, running behind Proxy,  to the Avro-agent, running on Linux box, and without Cygwin.. by just running JVM process. So happy.
>
> Cheers,
> AK47
>
>
> -----Original Message-----
> From: Alexander Lorenz [mailto:[EMAIL PROTECTED]]
> Sent: Monday, December 17, 2012 2:58 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Flume to stream logs live
>
> Hey,
>
> fluentd is a good tip, they have a custom sink:
> https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Plugins
>
> This could work.
>
> Best,
> Alex
>
> On Dec 17, 2012, at 2:33 AM, Juhani Connolly <[EMAIL PROTECTED]> wrote:
>
>> If you can't get flume running on windows your alternatives look like syslog, scribed and fluentd. We used scribed before moving to flume, but tbh, it can be a pain to compile and the hdfs support on it is not the best, we had data loss if there was an hdfs restart etc. I can't say much about the other two
>>
>> On 12/14/2012 11:47 PM, Kartashov, Andy wrote:
>>>
>>> Flummers,
>>>
>>> Loved working with Flume 1.2 -- very easy and simple configuration, it was a pleasure to work with. Managed to "tail --F" logs from unix server and into a hdfs cluster. The problem started when I also needed to push logs from a Windows application server.  Spent three days researching on how to install flume on Windows and run  a deamon/agent that will push the logs to the Avro source I successfully configured and ran on Unix. No luck. So I am looking t alternative. Is there other framework available out there to help me with my issue. What about scribe?
>>>
>
>>>
>
>>>
>>> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
>>
>
> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

Thank you guys.

Managed to install flume-ng  and run flume agent on windows at CMD line using java VM, i.e C:> java -Dlog4j.configuration= .... -cp "...\lib\*" -f "flume.conf" -n agent-test

Having source - seq / and sink - logger appeared to be working.
Having source - syslogTcp / and sink - logger appeared to be working, or did it?? See below/ Every second lifestatusSupervisor would advise : Status ceck complete.  What's the next step.
Can you actually send data to a certain port. Can you try running the flume client or execute bash -tail command somehow in Windows without running cygwin?
[cid:[EMAIL PROTECTED]BEA0]
From: Juhani Connolly [mailto:[EMAIL PROTECTED]]
Sent: Sunday, December 16, 2012 8:33 PM
To: [EMAIL PROTECTED]
Subject: Re: Flume to stream logs live

If you can't get flume running on windows your alternatives look like syslog, scribed and fluentd. We used scribed before moving to flume, but tbh, it can be a pain to compile and the hdfs support on it is not the best, we had data loss if there was an hdfs restart etc. I can't say much about the other two

On 12/14/2012 11:47 PM, Kartashov, Andy wrote:
Flummers,

Loved working with Flume 1.2 - very easy and simple configuration, it was a pleasure to work with. Managed to "tail -F" logs from unix server and into a hdfs cluster. The problem started when I also needed to push logs from a Windows application server.  Spent three days researching on how to install flume on Windows and run  a deamon/agent that will push the logs to the Avro source I successfully configured and ran on Unix. No luck. So I am looking t alternative. Is there other framework available out there to help me with my issue. What about scribe?

Andy Kartashov
MPAC
IT Architecture, Co-op
1340 Pickering Parkway, Pickering, L1V 0C4
* Phone : (905) 837 6269
* Mobile: (416) 722 1787
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

Hi,

FWIW, I was sending log data from Windows I would write a little
Windows Log Agent and send the data to the HTTP Source.

Brock

On Fri, Dec 14, 2012 at 8:47 AM, Kartashov, Andy <[EMAIL PROTECTED]> wrote:
> Flummers,
>
>
>
> Loved working with Flume 1.2 – very easy and simple configuration, it was a
> pleasure to work with. Managed to “tail –F” logs from unix server and into a
> hdfs cluster. The problem started when I also needed to push logs from a
> Windows application server.  Spent three days researching on how to install
> flume on Windows and run  a deamon/agent that will push the logs to the Avro
> source I successfully configured and ran on Unix. No luck. So I am looking t
> alternative. Is there other framework available out there to help me with my
> issue. What about scribe?
>
>
>
> Andy Kartashov
>
> MPAC
>
> IT Architecture, Co-op
>
> 1340 Pickering Parkway, Pickering, L1V 0C4
>
> ( Phone : (905) 837 6269
>
> ( Mobile: (416) 722 1787
>
> [EMAIL PROTECTED]
>
>
>
> NOTICE: This e-mail message and any attachments are confidential, subject to
> copyright and may be privileged. Any unauthorized use, copying or disclosure
> is prohibited. If you are not the intended recipient, please delete and
> contact the sender immediately. Please consider the environment before
> printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui
> l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent
> être couverts par le secret professionnel. Toute utilisation, copie ou
> divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire
> prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur.
> Veuillez penser à l'environnement avant d'imprimer le présent courriel

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

I have setup a Windows flume flow using LogParser and the AvroClient app bundled with flume.

It's a Powershell script scheduled every 5 minutes which runs a checkpointed query via LogParser to create incremental files for IIS logs and a couple other of our app logs. Then the incremental files are sent to a flume node running AvroSource. From there it's a typical flume setup, the log types are split based on a header that I append when sending via the AvroClient and then sent to collector nodes that sink to HDFS.

It's currently a best effort architecture as I don't trap any errors from the AvroClient on the Windows side. I did extend the AvroClient to kick out exit codes though, just not using it yet (see https://issues.apache.org/jira/browse/FLUME-1670). I've been sending about 15GB of IIS logs per day per server without issues, though.

It's not the best solution but it works for now. Longer term we are thinking of a custom app on our side that leverages the HTTPSource, or if we get ambitious implementing the AvroRPC in .net but that's a backburner project right now.

Also, I'm bucketing the events based on a timestamp interceptor which has caused post processing pain as the event timestamps are off by ~5 minutes from the header. I'm looking forward to using regex capture interceptor to timestamp the events with the event time soon.

Thanks,
Paul Chavez

-----Original Message-----
From: Brock Noland [mailto:[EMAIL PROTECTED]]
Sent: Friday, December 14, 2012 6:52 AM
To: [EMAIL PROTECTED]
Subject: Re: Flume to stream logs live

Hi,

FWIW, I was sending log data from Windows I would write a little Windows Log Agent and send the data to the HTTP Source.

Brock

On Fri, Dec 14, 2012 at 8:47 AM, Kartashov, Andy <[EMAIL PROTECTED]> wrote:
> Flummers,
>
>
>
> Loved working with Flume 1.2 - very easy and simple configuration, it
> was a pleasure to work with. Managed to "tail -F" logs from unix
> server and into a hdfs cluster. The problem started when I also needed
> to push logs from a Windows application server.  Spent three days
> researching on how to install flume on Windows and run  a deamon/agent
> that will push the logs to the Avro source I successfully configured
> and ran on Unix. No luck. So I am looking t alternative. Is there
> other framework available out there to help me with my issue. What about scribe?
>
>
>
> Andy Kartashov
>
> MPAC
>
> IT Architecture, Co-op
>
> 1340 Pickering Parkway, Pickering, L1V 0C4
>
> ( Phone : (905) 837 6269
>
> ( Mobile: (416) 722 1787
>
> [EMAIL PROTECTED]
>
>
>
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use,
> copying or disclosure is prohibited. If you are not the intended
> recipient, please delete and contact the sender immediately. Please
> consider the environment before printing this e-mail. AVIS : le
> présent courriel et toute pièce jointe qui l'accompagne sont
> confidentiels, protégés par le droit d'auteur et peuvent être couverts
> par le secret professionnel. Toute utilisation, copie ou divulgation
> non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur.
> Veuillez penser à l'environnement avant d'imprimer le présent courriel

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

You could also use syslog (or its variants syslog-ng, rsyslog) as a bridge
before flume. I don't know the details but I know there are syslog
implementations for windows too.

Regards

Bertrand

On Fri, Dec 14, 2012 at 5:24 PM, Paul Chavez <
[EMAIL PROTECTED]> wrote:

> I have setup a Windows flume flow using LogParser and the AvroClient app
> bundled with flume.
>
> It's a Powershell script scheduled every 5 minutes which runs a
> checkpointed query via LogParser to create incremental files for IIS logs
> and a couple other of our app logs. Then the incremental files are sent to
> a flume node running AvroSource. From there it's a typical flume setup, the
> log types are split based on a header that I append when sending via the
> AvroClient and then sent to collector nodes that sink to HDFS.
>
> It's currently a best effort architecture as I don't trap any errors from
> the AvroClient on the Windows side. I did extend the AvroClient to kick out
> exit codes though, just not using it yet (see
> https://issues.apache.org/jira/browse/FLUME-1670). I've been sending
> about 15GB of IIS logs per day per server without issues, though.
>
> It's not the best solution but it works for now. Longer term we are
> thinking of a custom app on our side that leverages the HTTPSource, or if
> we get ambitious implementing the AvroRPC in .net but that's a backburner
> project right now.
>
> Also, I'm bucketing the events based on a timestamp interceptor which has
> caused post processing pain as the event timestamps are off by ~5 minutes
> from the header. I'm looking forward to using regex capture interceptor to
> timestamp the events with the event time soon.
>
> Thanks,
> Paul Chavez
>
> -----Original Message-----
> From: Brock Noland [mailto:[EMAIL PROTECTED]]
> Sent: Friday, December 14, 2012 6:52 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Flume to stream logs live
>
> Hi,
>
> FWIW, I was sending log data from Windows I would write a little Windows
> Log Agent and send the data to the HTTP Source.
>
> Brock
>
> On Fri, Dec 14, 2012 at 8:47 AM, Kartashov, Andy <[EMAIL PROTECTED]>
> wrote:
> > Flummers,
> >
> >
> >
> > Loved working with Flume 1.2 - very easy and simple configuration, it
> > was a pleasure to work with. Managed to "tail -F" logs from unix
> > server and into a hdfs cluster. The problem started when I also needed
> > to push logs from a Windows application server.  Spent three days
> > researching on how to install flume on Windows and run  a deamon/agent
> > that will push the logs to the Avro source I successfully configured
> > and ran on Unix. No luck. So I am looking t alternative. Is there
> > other framework available out there to help me with my issue. What about
> scribe?
> >
> >
> >
> > Andy Kartashov
> >
> > MPAC
> >
> > IT Architecture, Co-op
> >
> > 1340 Pickering Parkway, Pickering, L1V 0C4
> >
> > ( Phone : (905) 837 6269
> >
> > ( Mobile: (416) 722 1787
> >
> > [EMAIL PROTECTED]
> >
> >
> >
> > NOTICE: This e-mail message and any attachments are confidential,
> > subject to copyright and may be privileged. Any unauthorized use,
> > copying or disclosure is prohibited. If you are not the intended
> > recipient, please delete and contact the sender immediately. Please
> > consider the environment before printing this e-mail. AVIS : le
> > présent courriel et toute pièce jointe qui l'accompagne sont
> > confidentiels, protégés par le droit d'auteur et peuvent être couverts
> > par le secret professionnel. Toute utilisation, copie ou divulgation
> > non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de
> ce courriel, supprimez-le et contactez immédiatement l'expéditeur.
> > Veuillez penser à l'environnement avant d'imprimer le présent courriel
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

--
Bertrand Dechoux

Hello Andy,

    You can visit this nice
post<http://mapredit.blogspot.in/2012/07/run-flume-13x-on-windows.html>by
Alex which describes how to use Flume on Windows.

Regards,
    Mohammad Tariq

On Fri, Dec 14, 2012 at 8:17 PM, Kartashov, Andy <[EMAIL PROTECTED]>wrote:

>  Flummers,
>
>
>
> Loved working with Flume 1.2 – very easy and simple configuration, it was
> a pleasure to work with. Managed to “tail –F” logs from unix server and
> into a hdfs cluster. The problem started when I also needed to push logs
> from a Windows application server.  Spent three days researching on how to
> install flume on Windows and run  a deamon/agent that will push the logs to
> the Avro source I successfully configured and ran on Unix. No luck. So I am
> looking t alternative. Is there other framework available out there to help
> me with my issue. What about scribe?
>
>
>
> Andy Kartashov
>
> *MPAC*
>
> IT Architecture, Co-op
>
> 1340 Pickering Parkway, Pickering, L1V 0C4
>
> ( Phone : (905) 837 6269
>
> ( Mobile: (416) 722 1787
>
> *[EMAIL PROTECTED]*
>
>
>  NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>

Alex,

This is the overview on what I have done.

Windows
 (running Flume agent inside JVM.  in cmd.exe:  java log4J.configuration=.... -cp "......\lib\*" org.apache.flume.node.Application -f ... -n... )
source: exec running tail command
sink: avro bind to a localhost/port

putty configured to pipe from local host/port  to remote host/port

EC2 AMI
(running Flume agent script:$  flume-ng agent -f.. -n .....)
Source: avro, listening on the port mentioned above
Sink: hdfs

Every time I would append something to a log file on Windows box, it will get pushed and stored  in hdfs.

Rgds,
-----Original Message-----
From: Alexander Alten-Lorenz [mailto:[EMAIL PROTECTED]]
Sent: Thursday, December 20, 2012 2:03 AM
To: [EMAIL PROTECTED]
Subject: Re: Flume to stream logs live

Hey AK47,

good to know, can you describe for the community what exactly you've done and what you've archived? Would be great,

cheers,
 Alex

On Dec 19, 2012, at 5:20 PM, "Kartashov, Andy" <[EMAIL PROTECTED]> wrote:

> Thank you for your suggestions: Mr. Chavez , Mr. Lorenz, Mr. Dechoux and Mr. Connolly,
>
> I just managed to successfully configure flume to push logs from windows box, running behind Proxy,  to the Avro-agent, running on Linux box, and without Cygwin.. by just running JVM process. So happy.
>
> Cheers,
> AK47
>
>
> -----Original Message-----
> From: Alexander Lorenz [mailto:[EMAIL PROTECTED]]
> Sent: Monday, December 17, 2012 2:58 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Flume to stream logs live
>
> Hey,
>
> fluentd is a good tip, they have a custom sink:
> https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Plugins
>
> This could work.
>
> Best,
> Alex
>
> On Dec 17, 2012, at 2:33 AM, Juhani Connolly <[EMAIL PROTECTED]> wrote:
>
>> If you can't get flume running on windows your alternatives look like syslog, scribed and fluentd. We used scribed before moving to flume, but tbh, it can be a pain to compile and the hdfs support on it is not the best, we had data loss if there was an hdfs restart etc. I can't say much about the other two
>>
>> On 12/14/2012 11:47 PM, Kartashov, Andy wrote:
>>>
>>> Flummers,
>>>
>>> Loved working with Flume 1.2 -- very easy and simple configuration, it was a pleasure to work with. Managed to "tail --F" logs from unix server and into a hdfs cluster. The problem started when I also needed to push logs from a Windows application server.  Spent three days researching on how to install flume on Windows and run  a deamon/agent that will push the logs to the Avro source I successfully configured and ran on Unix. No luck. So I am looking t alternative. Is there other framework available out there to help me with my issue. What about scribe?
>>>
>
>>>
>
>>>
>>> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
>>
>
> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel