Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> .SpoolingFileLineReader warning....


Copy link to this message
-
Re: .SpoolingFileLineReader warning....
Just a quick followup on this....I moved all the logging, etc to the same
drive/device and I'm no longer seeing those permission errors!.   I'm going
to keep testing this week.

Thank you Brock.

Regards,

Dano
On Tue, Nov 20, 2012 at 10:14 AM, Brock Noland <[EMAIL PROTECTED]> wrote:

> OK yeah, so basically it's doing a copy and then a delete. See "man 2
> rename" then ERRORS -> EXDEV but basically atomically renaming a file
> only works on the same mount point.
>
> On Tue, Nov 20, 2012 at 11:10 AM, Dan Young <[EMAIL PROTECTED]> wrote:
> > This is running on an EC2 instance, and these are the ephemeral drive(s).
> >
> > Filesystem     1K-blocks    Used Available Use% Mounted on
> > /dev/xvda1       8256952 1618248   6219276  21% /
> > udev             1912272       8   1912264   1% /dev
> > tmpfs             768100     180    767920   1% /run
> > none                5120       0      5120   0% /run/lock
> > none             1920244       0   1920244   0% /run/shm
> > /dev/xvdb      412814512 1127172 390717564   1% /mnt
> >
> > ubuntu@ip-foo:~$ cat /etc/fstab
> > LABEL=cloudimg-rootfs / ext4 defaults 0 0
> > /dev/xvdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2
> > ubuntu@ip-foo:~$
> >
> > ubuntu@ip-foo:~$ mount
> > /dev/xvda1 on / type ext4 (rw)
> > proc on /proc type proc (rw,noexec,nosuid,nodev)
> > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> > none on /sys/fs/fuse/connections type fusectl (rw)
> > none on /sys/kernel/debug type debugfs (rw)
> > none on /sys/kernel/security type securityfs (rw)
> > udev on /dev type devtmpfs (rw,mode=0755)
> > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> > none on /run/shm type tmpfs (rw,nosuid,nodev)
> > /dev/xvdb on /mnt type ext3 (rw)
> >
> > I do see that / is ext4 and /mnt is ext3.....hmm, not sure it matters.
> >
> > Dano
> >
> >
> >
> > On Tue, Nov 20, 2012 at 10:01 AM, Brock Noland <[EMAIL PROTECTED]>
> wrote:
> >>
> >> Are /var/log and /mnt/flume on the same file system? mv across file
> >> systems is a cp and then delete.
> >>
> >> On Tue, Nov 20, 2012 at 10:59 AM, Dan Young <[EMAIL PROTECTED]>
> wrote:
> >> > Coolio, thank you Brock.
> >> >
> >> > Did a quick test with a mv vs cp.....this is one test, repeated a few
> >> > times....the smaller file (~60M) seemed to work fine w/ a mv, but a
> >> > larger
> >> > file we're seeing the same behavior....
> >> >
> >> >
> >> > Start with no logs in the SpoolingDirectory:
> >> >
> >> > ls -lrt /mnt/flume/clickstream
> >> >
> >> > /mnt/flume/clickstream:
> >> > total 0
> >> >
> >> >
> >> >
> >> > Review the logs that will be rotated via logrotate.d into the
> respective
> >> > directory; /mnt/flume/clickstream
> >> >
> >> > ls -lrt /var/log/clickstream
> >> > /var/log/clickstream:
> >> > total 64112
> >> > -rw-rw-r-- 1 ubuntu ubuntu 65648336 Nov 20 16:05 clickstream.log
> >> >
> >> >
> >> >
> >> > Review logrotate config in /etc/logrotate.d. Note here, I changed from
> >> > cp -p
> >> > to a mv.....
> >> >
> >> > /var/log/clickstream/clickstream.log
> >> > {
> >> >   missingok
> >> >   rotate 3
> >> >   compress
> >> >   delaycompress
> >> >   copytruncate
> >> >   notifempty
> >> >   size 50M
> >> >   dateext
> >> >   dateformat -%Y-%m-%d-%s
> >> >   create 666 ubuntu ubuntu
> >> >   postrotate
> >> >   mv $1 /mnt/flume/clickstream/ 2>&1
> >> >   endscript
> >> > }
> >> >
> >> >
> >> > I run logrotate.d/clickstream.POST with the ~60MB file, and everything
> >> > looked fine....now I try a ~190MB file....
> >> >
> >> >
> >> >
> >> > ls -lrt /var/log/clickstream
> >> > /var/log/clickstream:
> >> > total 192336
> >> > -rw-rw-r-- 1 ubuntu ubuntu 196945008 Nov 20 16:42 clickstream.log
> >> >
> >> >
> >> > Run logrotate.d/clickstream.POST, and we see the WARNING in the FLume
> >> > log.
> >> >
> >> > ....
> >> > ....
> >> > 20 Nov 2012 16:45:07,117 WARN  [pool-13-thread-1]