Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Controlled shutdown failure, retry settings


Copy link to this message
-
Re: Controlled shutdown failure, retry settings
On Fri, Oct 25, 2013 at 9:16 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

>
> Unclean shutdown could result in data loss - since you are moving
> leadership to a replica that has fallen out of ISR. i.e., it's log end
> offset is behind the last committed message to this partition.
>
>
But if data is written with 'request.required.acks=-1', no data should be
lost, no?  Or will partitions be truncated wholesale after an unclean
shutdown?

>
> Take a look at the packaged log4j.properties file. The controller's
> partition/replica state machines and its channel manager which
> sends/receives leaderandisr requests/responses to brokers uses a
> stateChangeLogger. The replica managers on all brokers also use this
> logger.
Ah.....so it looks like most things logged with the stateChangeLogger are
logged at the TRACE log level.....and that's the default setting in the
log4j.properties file.  Needless to say, my contained KafkaServer is not
currently using that log4j.properties (we are just using a rootLogger with
level = INFO by default).  I can probably add a special rule to use TRACE
for the state.change.logger category.  However, I'm not sure I can make it
so that logging all goes to it's own separate log file.....

>
> Our logging can improve - e.g., it looks like on controller movement
> we could retry without saying why.
>

I can file a jira for this, but I'm not sure what it should say!

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB