Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> having problem with 0.8 gzip compression


+
Scott Wang 2013-07-08, 21:54
+
Jun Rao 2013-07-09, 04:07
+
Scott Wang 2013-07-09, 04:38
+
Jun Rao 2013-07-09, 15:07
+
Scott Wang 2013-07-09, 18:08
+
Scott Wang 2013-07-09, 22:23
Copy link to this message
-
Re: having problem with 0.8 gzip compression
Could you run the following command on one of the log files of your topic
and attach the output?

bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files
/tmp/kafka-logs/testtopic-0/00000000000000000000.log

Thanks,

Jun
On Tue, Jul 9, 2013 at 3:23 PM, Scott Wang <
[EMAIL PROTECTED]> wrote:

> Another piece of information, the snappy compression also does not work.
>
> Thanks,
> Scott
>
>
> On Tue, Jul 9, 2013 at 11:07 AM, Scott Wang <
> [EMAIL PROTECTED]> wrote:
>
> > I just try it and it still not showing up, thanks for looking into this.
> >
> > Thanks,
> > Scott
> >
> >
> > On Tue, Jul 9, 2013 at 8:06 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> Could you try starting the consumer first (and enable gzip in the
> >> producer)?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Mon, Jul 8, 2013 at 9:37 PM, Scott Wang <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >> > No, I did not start the consumer before the producer.  I actually
> >> started
> >> > the producer first and nothing showed up in the consumer unless I
> >> commented
> >> > out this line -- props.put("compression.codec", "gzip").    If I
> >> commented
> >> > out the compression codec, everything just works.
> >> >
> >> >
> >> > On Mon, Jul 8, 2013 at 9:07 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >> >
> >> > > Did you start the consumer before the producer? Be default, the
> >> consumer
> >> > > gets only the new data?
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > >
> >> > > On Mon, Jul 8, 2013 at 2:53 PM, Scott Wang <
> >> > > [EMAIL PROTECTED]> wrote:
> >> > >
> >> > > > I am testing with Kafka 0.8 beta and having problem of receiving
> >> > message
> >> > > in
> >> > > > consumer.  There is no error so does anyone have any insights.
> >>  When I
> >> > > > commented out the "compression.code" everything works fine.
> >> > > >
> >> > > > My producer:
> >> > > > public class TestKafka08Prod {
> >> > > >
> >> > > >     public static void main(String [] args) {
> >> > > >
> >> > > >         Producer<Integer, String> producer = null;
> >> > > >         try {
> >> > > >             Properties props = new Properties();
> >> > > >             props.put("metadata.broker.list", "localhost:9092");
> >> > > >             props.put("serializer.class",
> >> > > > "kafka.serializer.StringEncoder");
> >> > > >             props.put("producer.type", "sync");
> >> > > >             props.put("request.required.acks","1");
> >> > > >             props.put("compression.codec", "gzip");
> >> > > >             ProducerConfig config = new ProducerConfig(props);
> >> > > >             producer = new Producer<Integer, String>(config);
> >> > > >             int j=0;
> >> > > >             for(int i=0; i<10; i++) {
> >> > > >                 KeyedMessage<Integer, String> data = new
> >> > > > KeyedMessage<Integer, String>("test-topic", "test-message: "+i+"
> >> > > > "+System.currentTimeMillis());
> >> > > >                 producer.send(data);
> >> > > >
> >> > > >             }
> >> > > >
> >> > > >         } catch (Exception e) {
> >> > > >             System.out.println("Error happened: ");
> >> > > >             e.printStackTrace();
> >> > > >         } finally {
> >> > > >             if(null != null) {
> >> > > >                 producer.close();
> >> > > >             }
> >> > > >
> >> > > >             System.out.println("Ened of Sending");
> >> > > >         }
> >> > > >
> >> > > >         System.exit(0);
> >> > > >     }
> >> > > > }
> >> > > >
> >> > > >
> >> > > > My consumer:
> >> > > >
> >> > > > public class TestKafka08Consumer {
> >> > > >     public static void main(String [] args) throws
> >> > UnknownHostException,
> >> > > > SocketException {
> >> > > >
> >> > > >         Properties props = new Properties();
> >> > > >         props.put("zookeeper.connect",
> "localhost:2181/kafka_0_8");
> >> > > >         props.put("group.id", "test08ConsumerId");
> >> > > >         props.put("zk.sessiontimeout.ms", "4000");

 
+
Scott Wang 2013-07-10, 17:24
+
Joel Koshy 2013-07-11, 00:33
+
Scott Wang 2013-07-11, 18:42
+
Scott Wang 2013-07-11, 23:02