Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> where does the in-memory log to file persistence take place


Copy link to this message
-
Re: where does the in-memory log to file persistence take place
couldn't this be written in a way that it doesn't block? or limits the
time, like it makes a copy of the reference, then replaces it when a newed
up channel (synchronized).

Oh, this isn't possible because the var is mapped to a file at the o/s
level?

On Tue, May 8, 2012 at 5:38 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> filechannel.force() always fully syncs the file to disk. This is done
> irrespective of message boundaries. The file is locked during this
> time so other appends are blocked.
>
> -Jay
>
> On Tue, May 8, 2012 at 1:44 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > Slowly trying to understand it, have to wramp up on my scala.
> >
> > When the flush/sink occurrs, does it pull items of the collection 1 by 1
> or
> > does it do this in bulk somehow while locking the collection?
> >
> > On Mon, May 7, 2012 at 3:14 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> >> Ahmed,
> >>
> >> The related code is in kafka.log.*. The message to file persistence is
> >> inside FileMessageSet.scala.
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Mon, May 7, 2012 at 12:12 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> >>
> >> > I can barely read scala, but I'm curious where the applications
> performs
> >> > the operation of taking the in-memory log and persisting it to the
> >> > database, all the while accepting new log messages and removing the
> keys
> >> of
> >> > the messages that have been persisted to disk.
> >> >
> >> > I'm guessing you have used the concurrenthashmap where the key is a
> >> topic,
> >> > and once the flush timeout has been reached a background thread will
> >> >  somehow persist and remove the keys.
> >> >
> >>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB