Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> where does the in-memory log to file persistence take place


Copy link to this message
-
Re: where does the in-memory log to file persistence take place
couldn't this be written in a way that it doesn't block? or limits the
time, like it makes a copy of the reference, then replaces it when a newed
up channel (synchronized).

Oh, this isn't possible because the var is mapped to a file at the o/s
level?

On Tue, May 8, 2012 at 5:38 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> filechannel.force() always fully syncs the file to disk. This is done
> irrespective of message boundaries. The file is locked during this
> time so other appends are blocked.
>
> -Jay
>
> On Tue, May 8, 2012 at 1:44 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > Slowly trying to understand it, have to wramp up on my scala.
> >
> > When the flush/sink occurrs, does it pull items of the collection 1 by 1
> or
> > does it do this in bulk somehow while locking the collection?
> >
> > On Mon, May 7, 2012 at 3:14 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> >> Ahmed,
> >>
> >> The related code is in kafka.log.*. The message to file persistence is
> >> inside FileMessageSet.scala.
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Mon, May 7, 2012 at 12:12 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> >>
> >> > I can barely read scala, but I'm curious where the applications
> performs
> >> > the operation of taking the in-memory log and persisting it to the
> >> > database, all the while accepting new log messages and removing the
> keys
> >> of
> >> > the messages that have been persisted to disk.
> >> >
> >> > I'm guessing you have used the concurrenthashmap where the key is a
> >> topic,
> >> > and once the flush timeout has been reached a background thread will
> >> >  somehow persist and remove the keys.
> >> >
> >>
>