Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.


Copy link to this message
-
Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.
Guozhang Wang 2013-08-28, 17:11
I think this patch can be made in trunk. You can mark it as 0.8.1

Guozhang

On Wednesday, August 28, 2013, Rajasekar Elango <[EMAIL PROTECTED]>
wrote:
> Guozhang ,
>
> *The documentation says I need to work off of trunk. Can you confirm If I
> should be working in trunk or different branch.*
> *
> *
> *Thanks,*
> *Raja.*
>
>
> On Tue, Aug 27, 2013 at 8:33 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
>
>> Cool! You can follow the process of creating a JIRA here:
>>
>> http://kafka.apache.org/contributing.html
>>
>> And submit patch here:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Git+Workflow
>>
>> It will be great if you can also add an entry for this issue in FAQ
since I
>> think this is a common question:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/FAQ
>>
>> Guozhang
>>
>>
>> On Tue, Aug 27, 2013 at 2:38 PM, Rajasekar Elango <[EMAIL PROTECTED]
>> >wrote:
>>
>> > Thanks Guozhang, Changing max retry to 5 worked. Since I am changing
>> > console producer code, I can also submit patch adding both
>> > message.send.max.retries
>> > and retry.backoff.ms to console producer. Can you let me know process
>> for
>> > submitting patch?
>> >
>> > Thanks,
>> > Raja.
>> >
>> >
>> > On Tue, Aug 27, 2013 at 4:03 PM, Guozhang Wang <[EMAIL PROTECTED]>
>> wrote:
>> >
>> > > Hello Rajasekar,
>> > >
>> > > The remove fetcher log entry is normal under addition of partitions,
>> > since
>> > > they indicate that some leader changes have happened so brokers are
>> > closing
>> > > the fetchers to the old leaders.
>> > >
>> > > I just realized that the console Producer does not have the
>> > > message.send.max.retries options yet. Could you file a JIRA for this
>> and
>> > I
>> > > will followup to add this option? As for now you can hard modify the
>> > > default value from 3 to a larger number.
>> > >
>> > > Guozhang
>> > >
>> > >
>> > > On Tue, Aug 27, 2013 at 12:37 PM, Rajasekar Elango
>> > > <[EMAIL PROTECTED]>wrote:
>> > >
>> > > > Thanks Neha & Guozhang,
>> > > >
>> > > > When I ran StateChangeLogMerger, I am seeing this message repeated
16
>> > > times
>> > > > for each partition:
>> > > >
>> > > > [2013-08-27 12:30:02,535] INFO [ReplicaFetcherManager on broker 1]
>> > > Removing
>> > > > fetcher for partition [test-60,13]
>> (kafka.server.ReplicaFetcherManager)
>> > > > [2013-08-27 12:30:02,536] INFO [Log Manager on Broker 1] Created
log
>> > for
>> > > > partition [test-60,13] in
>> > > >
>> /home/relango/dev/mandm/kafka/main/target/dist/mandm-kafka/kafka-data.
>> > > > (kafka.log.LogManager)
>> > > >
>> > > > I am also seeing .log and .index files created for this topic in
data
>> > > dir.
>> > > > Also list topic command shows leaders, replicas and isrs for all
>> > > > partitions. Do you still think increasing num of retries would help
>> or
>> > is
>> > > > it some other issue..? Also console Producer doesn't seem to  have
>> > option
>> > > > to set num of retries. Is there a way to configure num of retries
for
>> > > > console producer ?
>> > > >
>> > > > Thanks,
>> > > > Raja.
>> > > >
>> > > >
>> > > > On Tue, Aug 27, 2013 at 12:52 PM, Neha Narkhede <
>> > [EMAIL PROTECTED]
>> > > > >wrote:
>> > > >
>> > > > > As Guozhang said, your producer might give up sooner than the
>> leader
>> > > > > election completes for the new topic. To confirm if your producer
>> > gave
>> > > up
>> > > > > too soon, you can run the state--
> Thanks,
> Raja.
>

--
-- Guozhang