Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> compression on by default?


+
Paul Garner 2012-11-22, 15:27
+
Neha Narkhede 2012-11-22, 18:45
+
Paul Garner 2012-11-22, 20:29
Copy link to this message
-
Re: compression on by default?
This seems to be a bug in ConsoleProducer. It also compresses messages and
ignores the --compress flag. Could you file a jira?

Thanks,

Jun

On Thu, Nov 22, 2012 at 12:29 PM, Paul Garner <[EMAIL PROTECTED]
> wrote:

> Neha,
>
> Thanks for taking the time to answer my questions.
>
> The only producer.properties file I can find is at:
> /home/ubuntu/kafka-0.7.2-incubating-src/config/producer.properties
>
> In there is:
> compression.codec=0
>
> My process looks like:
>
> root      1748  1746  0 Nov19 ?        00:02:37 java -Xmx512M -server
> -Dlog4j.configuration=file:/usr/local/bin/kafka/../config/log4j.properties
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false -cp
>
> :/usr/local/bin/kafka/../project/boot/scala-2.8.0/lib/scala-compiler.jar:/usr/local/bin/kafka/../project/boot/scala-2.8.0/lib/scala-library.jar:/usr/local/bin/kafka/../core/target/scala_2.8.0/kafka-0.7.2.jar:/usr/local/bin/kafka/../core/lib/*.jar:/usr/local/bin/kafka/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar
> kafka.producer.ConsoleProducer --topic logtail --zookeeper x.x.x.x:2181
>
> But the messages come out gobbledegook unless I use a client that
> understands compressed messages, and in that client it identifies the bit
> as set to 1, gzip compression.
>
> On Thu, Nov 22, 2012 at 6:45 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > The producer shell takes in a properties file. The default one that
> > comes with Kafka (config/producer.properties) turns compression off.
> > Can you cross check the value for compression.codec in your producer's
> > properties file ?
> >
> > Thanks,
> > Neha
> >
> > On Thu, Nov 22, 2012 at 7:27 AM, Paul Garner
> > <[EMAIL PROTECTED]> wrote:
> > > I am using the kafka-proiducer-shell.sh script without the --compress
> > option
> > >
> > > however my messages seem to be gzipped
> > >
> > > the docs say compression is off by default:
> > > http://incubator.apache.org/kafka/configuration.html
> > >
> > > is it on by default now in 0.7.2?
> >
>
+
Paul Garner 2012-11-27, 18:13
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB