Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - .SpoolingFileLineReader warning....


Copy link to this message
-
Re: .SpoolingFileLineReader warning....
Brock Noland 2012-11-20, 20:06
Great to hear it's working for you!

On Tue, Nov 20, 2012 at 2:03 PM, Dan Young <[EMAIL PROTECTED]> wrote:
> Just a quick followup on this....I moved all the logging, etc to the same
> drive/device and I'm no longer seeing those permission errors!.   I'm going
> to keep testing this week.
>
> Thank you Brock.
>
> Regards,
>
> Dano
>
>
> On Tue, Nov 20, 2012 at 10:14 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
>>
>> OK yeah, so basically it's doing a copy and then a delete. See "man 2
>> rename" then ERRORS -> EXDEV but basically atomically renaming a file
>> only works on the same mount point.
>>
>> On Tue, Nov 20, 2012 at 11:10 AM, Dan Young <[EMAIL PROTECTED]> wrote:
>> > This is running on an EC2 instance, and these are the ephemeral
>> > drive(s).
>> >
>> > Filesystem     1K-blocks    Used Available Use% Mounted on
>> > /dev/xvda1       8256952 1618248   6219276  21% /
>> > udev             1912272       8   1912264   1% /dev
>> > tmpfs             768100     180    767920   1% /run
>> > none                5120       0      5120   0% /run/lock
>> > none             1920244       0   1920244   0% /run/shm
>> > /dev/xvdb      412814512 1127172 390717564   1% /mnt
>> >
>> > ubuntu@ip-foo:~$ cat /etc/fstab
>> > LABEL=cloudimg-rootfs / ext4 defaults 0 0
>> > /dev/xvdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2
>> > ubuntu@ip-foo:~$
>> >
>> > ubuntu@ip-foo:~$ mount
>> > /dev/xvda1 on / type ext4 (rw)
>> > proc on /proc type proc (rw,noexec,nosuid,nodev)
>> > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
>> > none on /sys/fs/fuse/connections type fusectl (rw)
>> > none on /sys/kernel/debug type debugfs (rw)
>> > none on /sys/kernel/security type securityfs (rw)
>> > udev on /dev type devtmpfs (rw,mode=0755)
>> > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
>> > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
>> > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
>> > none on /run/shm type tmpfs (rw,nosuid,nodev)
>> > /dev/xvdb on /mnt type ext3 (rw)
>> >
>> > I do see that / is ext4 and /mnt is ext3.....hmm, not sure it matters.
>> >
>> > Dano
>> >
>> >
>> >
>> > On Tue, Nov 20, 2012 at 10:01 AM, Brock Noland <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> Are /var/log and /mnt/flume on the same file system? mv across file
>> >> systems is a cp and then delete.
>> >>
>> >> On Tue, Nov 20, 2012 at 10:59 AM, Dan Young <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Coolio, thank you Brock.
>> >> >
>> >> > Did a quick test with a mv vs cp.....this is one test, repeated a few
>> >> > times....the smaller file (~60M) seemed to work fine w/ a mv, but a
>> >> > larger
>> >> > file we're seeing the same behavior....
>> >> >
>> >> >
>> >> > Start with no logs in the SpoolingDirectory:
>> >> >
>> >> > ls -lrt /mnt/flume/clickstream
>> >> >
>> >> > /mnt/flume/clickstream:
>> >> > total 0
>> >> >
>> >> >
>> >> >
>> >> > Review the logs that will be rotated via logrotate.d into the
>> >> > respective
>> >> > directory; /mnt/flume/clickstream
>> >> >
>> >> > ls -lrt /var/log/clickstream
>> >> > /var/log/clickstream:
>> >> > total 64112
>> >> > -rw-rw-r-- 1 ubuntu ubuntu 65648336 Nov 20 16:05 clickstream.log
>> >> >
>> >> >
>> >> >
>> >> > Review logrotate config in /etc/logrotate.d. Note here, I changed
>> >> > from
>> >> > cp -p
>> >> > to a mv.....
>> >> >
>> >> > /var/log/clickstream/clickstream.log
>> >> > {
>> >> >   missingok
>> >> >   rotate 3
>> >> >   compress
>> >> >   delaycompress
>> >> >   copytruncate
>> >> >   notifempty
>> >> >   size 50M
>> >> >   dateext
>> >> >   dateformat -%Y-%m-%d-%s
>> >> >   create 666 ubuntu ubuntu
>> >> >   postrotate
>> >> >   mv $1 /mnt/flume/clickstream/ 2>&1
>> >> >   endscript
>> >> > }
>> >> >
>> >> >
>> >> > I run logrotate.d/clickstream.POST with the ~60MB file, and
>> >> > everything
>> >> > looked fine....now I try a ~190MB file....
>> >> >
>> >> >
>> >> >
>> >> > ls -lrt /var/log/clickstream
>> >> > /var/log/clickstream:
>> >
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/