I am able to post messages with, but after certain days, kafka creating multiple log files, messages are posted in new log file, although old file size is very less.
What can be the reason for creating multiple log files? Does message format matters while sending message to kafka?
Snehalata Disclaimer: This e-mail may contain Privileged/Confidential information and is intended only for the individual(s) named. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. Please notify the sender, if you have received this e-mail by mistake and delete it from your system. Information in this message that does not relate to the official business of the company shall be understood as neither given nor endorsed by it. E-mail transmission cannot be guaranteed to be secure or error-free. The sender does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Visit us at http://www.harbingergroup.com/
Just curious - given the situation above, if you have two sets of .log and .index files under the same topic and same partition, is there any way to merge these two sets of log files and index files? On Thu, Apr 18, 2013 at 5:42 AM, Swapnil Ghike <[EMAIL PROTECTED]> wrote:
Number of open file handles is one thing. But I am thinking more of a use case where you may want to fetch information from two separate log files of same topic and same partition. Could we use one high level consumer API call to achieve that? One naive idea is that if we can merge those two log files (since they are of same topic and in partition I thought we might be able to merge them) into one, then using one high level consumer API will do the job. On Thu, Apr 18, 2013 at 11:26 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
I think there is a misunderstanding here. The log file division is completely transparent to the consumer. The consumer consumes by offset (0,1,2,3,...) and these offsets span log files (i.e. there is a single, monotonically increasing set of offsets over all log files in a partition). In other words consumers ALWAYS consume across log files based on the position (offset) in the set of all logs in the partition.
If you want fewer log files change the settings so that the log files are bigger. However the only impact of this is fewer file handles, it won't impact the consumer in any way at all.
Not sure if that helps... -Jay
On Thu, Apr 18, 2013 at 8:36 AM, Jason Huang <[EMAIL PROTECTED]> wrote:
Jason On Thu, Apr 18, 2013 at 12:20 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext