Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Unable to send and consume compressed events.


Copy link to this message
-
Re: Unable to send and consume compressed events.
Jun Rao 2013-08-30, 14:32
Xuechao,

Thanks for updating the wiki. Not setting up log4j properly seems to be a
general problem, not just limited to this particular problem. Could you
reword it a bit to make it more general?

Jun
On Fri, Aug 30, 2013 at 2:02 AM, Lu Xuechao <[EMAIL PROTECTED]> wrote:

> Hi, Joe. wiki updated. Hope it helps.
>
>
> On Fri, Aug 30, 2013 at 3:22 PM, Joe Stein <[EMAIL PROTECTED]> wrote:
>
> > I feel like this is maybe a usual case as we have heard it before now a
> few
> > bits
> >
> > Lu Xuechao would you mind updating the FAQ
> > https://cwiki.apache.org/confluence/display/KAFKA/FAQ with what the
> > problem
> > was and your solution just to capture this thread in the wiki please,
> > thanx!
> >
> > /*******************************************
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
> >
> > On Fri, Aug 30, 2013 at 2:30 AM, Lu Xuechao <[EMAIL PROTECTED]> wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for you help. Finally, I found the reason by enabling producer
> > side
> > > DEBUG info output. The snappy jar is not included in the classpath.
> Added
> > > it and it worked.
> > >
> > > Thanks again.
> > >
> > >
> > >
> > >
> > > On Fri, Aug 30, 2013 at 12:53 PM, Lu Xuechao <[EMAIL PROTECTED]> wrote:
> > >
> > > > No.
> > > >
> > > >
> > > > On Fri, Aug 30, 2013 at 11:57 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > >> These are the metadata requests. Do you see Producer requests from
> > your
> > > >> client?
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >>
> > > >> On Thu, Aug 29, 2013 at 5:40 PM, Lu Xuechao <[EMAIL PROTECTED]>
> wrote:
> > > >>
> > > >> > After I sent 1,000 compressed events, I saw these messages in
> > broker's
> > > >> log
> > > >> > files:
> > > >> >
> > > >> > in kafka-request.log
> > > >> >
> > > >> > [2013-08-30 08:38:18,713] TRACE Processor 6 received request :
> Name:
> > > >> > TopicMetadataRequest; Version: 0; CorrelationId: 0; ClientId: ;
> > > Topics:
> > > >> > topic1 (kafka.network.RequestChannel$)
> > > >> > [2013-08-30 08:38:18,718] TRACE Completed request:Name:
> > > >> > TopicMetadataRequest; Version: 0; CorrelationId: 0; ClientId: ;
> > > Topics:
> > > >> > topic1 from client
> > > >> > /127.0.0.1:64238
> > > >> > ;totalTime:5,queueTime:3,localTime:1,remoteTime:0,sendTime:1
> > > >> > (kafka.request.logger)
> > > >> >
> > > >> >
> > > >> > in server.log
> > > >> >
> > > >> > [2013-08-30 08:38:18,759] INFO Closing socket connection to /
> > > 127.0.0.1.
> > > >> > (kafka.network.Processor)
> > > >> >
> > > >> >
> > > >> > any ideas?  Thanks.
> > > >> >
> > > >> >
> > > >> > On Thu, Aug 29, 2013 at 10:28 PM, Jun Rao <[EMAIL PROTECTED]>
> wrote:
> > > >> >
> > > >> > > Did you see any error in the producer log? Did the broker
> receive
> > > the
> > > >> > > produce request (you can look at the request log in the broker)?
> > > >> > >
> > > >> > > Thanks,
> > > >> > >
> > > >> > > Jun
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Aug 29, 2013 at 6:29 AM, Lu Xuechao <[EMAIL PROTECTED]>
> > > wrote:
> > > >> > >
> > > >> > > > Let me post my test code here. I could see
> producer.send(data);
> > > >> > returned
> > > >> > > > with no error.
> > > >> > > >
> > > >> > > > public class TestProducer extends Thread {
> > > >> > > >     private final Producer<String, String> producer;
> > > >> > > >
> > > >> > > >     private final int m_events;
> > > >> > > >     private final int m_threadNumber;
> > > >> > > >
> > > >> > > >     private static String msg = StringUtils.rightPad("", 1000,
> > > '*');
> > > >> > > >
> > > >> > > >     public TestProducer(int threadNumber, int events) {
> > > >> > > >         m_threadNumber = threadNumber;
> > > >> > > >         m_events = events;
> > > >> > > >
> > > >> > > >         Properties props = new Properties();
> > > >> > > >         props.put("serializer.class",