Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> File Channel  performance and fsync


Copy link to this message
-
File Channel  performance and fsync
Hi

I am writing this on top of another thread where there was discussion on
"fsync lies" and
only file channel used fsync and not file sink. :

-- I tested the fsync performance on 2 machines  (On 1 machine I was
getting very good throughput
using file channel and on another almost 100 times slower with almost
same hardware configuration.)
using following code
#define PAGESIZE 4096

int main(int argc, char *argv[])
{

         char my_write_str[PAGESIZE];
         char my_read_str[PAGESIZE];
         char *read_filename= argv[1];
         int readfd,writefd;

         readfd = open(read_filename,O_RDONLY);
         writefd = open("written_file",O_WRONLY|O_CREAT,777);
         int len=lseek(readfd,0,2);
         lseek(readfd,0,0);
         int iterations = len/PAGESIZE;
         int i;
         struct timeval t0,t1;

        for(i=0;i<iterations;i++)
         {

                 read(readfd,my_read_str,PAGESIZE);
                 write(writefd,my_read_str,PAGESIZE);
*gettimeofday(&t0,0);**
**                fsync(writefd);**
**              gettimeofday(&t1,0);*
                 long elapsed = (t1.tv_sec-t0.tv_sec)*1000000 +
t1.tv_usec-t0.tv_usec;
                 printf("Elapsed time is= %ld \n",elapsed);
          }
         close(readfd);
         close(writefd);
}
-- As expected it requires typically 50000 microseconds for fsync to
complete on one machine and 200 microseconds
on another machine it took 290 microseconds to complete on an average.
So is machine with higher
performance is doing a 'fsync lie'?
i
-- If I have understood it clearly; "fsync lie" means the data is not
actually written to disk and it is in
some disk/controller buffer.  I) Now if disk loses power due to some
shutdown or any other disaster, data will
be lost. II) Can data be lost even without it ? (e.g. if it is keeping
data in some disk buffer and if fsync is being
invoked continuously then will that data can also  be lost? If only part
-I is true; then it can be acceptable
because probability of shutdown is usually less in production
environment. But if even II is true then there is a
problem.

-- But on the machine where disk doesn't lie performance of flume using
File channel is very low (I have seen it
maximum 100 KB/sec even with sufficient  DirectMemory allocation.) Does
anybody have stats about throughput
of file channel ? Is anybody getting better performance with file
channel (without fsync lies). What is the recommended
usage of it for an average scenario ? (Transferring files of few MBs to
HDFS sink continuously on typical hardware
(16 core processors, 16 GB RAM etc.)
Regards,
Jagadish

On 10/10/2012 11:30 PM, Brock Noland wrote:
> Hi,
>
> On Wed, Oct 10, 2012 at 11:22 AM, Jagadish Bihani
> <[EMAIL PROTECTED]> wrote:
>> Hi Brock
>>
>> I will surely look into 'fsync lies'.
>>
>> But as per my experiments I think "file channel" is causing the issue.
>> Because on those 2 machines (one with higher throughput and other with
>> lower)
>> I did following experiment:
>>
>> cat Source -memory channel - file sink.
>>
>> Now with this setup I got same throughput on both the machines. (around 3
>> MB/sec)
>> Now as I have used "File sink" it should also do "fsync" at some point of
>> time.
>> 'File Sink' and 'File Channel' both do disk writes.
>> So if there is differences in disk behaviour then even in the 'File Sink' it
>> should be visible.
>>
>> Am I missing something here?
> File sink does not call fsync.
>
>> Regards,
>> Jagadish
>>
>>
>>
>> On 10/10/2012 09:35 PM, Brock Noland wrote:
>>> OK your disk that is giving you 40KB/second is telling you the truth
>>> and the faster disk is lying to you. Look up "fsync lies" to see what
>>> I am referring to.
>>>
>>> A spinning disk can do 100 fsync operations per second (this is done
>>> at the end of every batch). That is how I estimated your event size,
>>> 40KB/second is doing 40KB / 100 =  409 bytes.
>>>
>>> Once again, if you want increased performance, you should increase the
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB