Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> Time delay question


Copy link to this message
-
RE: Time delay question
So the only tunable elements in the BatchWriter are memory size and
timeout?  Otherwise the data is available for the next scanner.  Thanks
for the help!

 

From: John Vines [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, August 08, 2012 14:06
To: [EMAIL PROTECTED]
Subject: Re: Time delay question

 

Once BatchWriter's return from flush() or when they automatically do a
flush in the background (when their buffers get sufficiently large), the
data is availble for scanners. At that point the data is at least in the
in memory map, as well as the walog.

As the in memory maps for the tablets for the table fill, they will
write to disk. And when there are more files of given sizes than
desired, major compaction will occur which will compact files to a file.

So when you scan, data is merged from the in memory map and the files.

John

On Wed, Aug 8, 2012 at 2:09 PM, Keith Turner <[EMAIL PROTECTED]> wrote:

Once the batch writer flushes, the data is available for read by the
batch scanner.   On the server side data is read from memory and
files.
On Wed, Aug 8, 2012 at 1:53 PM,  <[EMAIL PROTECTED]> wrote:
> I have a set of actors that are doing batchWriters and another set
doing
> batchScanner(s) and I'm getting questions from my users about the time
> delay between them.  So my question is, what are all the tunable
> parameters between writes and reads?
>
> Do I have the following facts correct?
>
> BatchWriters have a time value for flushing to memory cache
> Minor compaction has its time frequency for flushing to WALogs
> Major compaction has its time frequency for flushing to disc
>
> So where all does a batchScanner get its data from?
>
> Any/all help is appreciated.
>
>
> Bob Thorman
> Engineering Fellow
> L-3 Communications, ComCept
> 1700 Science Place
> Rockwall, TX 75032
> (972) 772-7501 <tel:%28972%29%20772-7501>  work
> [EMAIL PROTECTED]
> [EMAIL PROTECTED]
>
>