Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> .SpoolingFileLineReader warning....


+
Dan Young 2012-11-17, 15:02
+
Brock Noland 2012-11-17, 15:57
+
Dan Young 2012-11-17, 16:01
+
Brock Noland 2012-11-17, 16:15
+
Dan Young 2012-11-17, 16:33
+
Dan Young 2012-11-19, 19:33
+
Patrick Wendell 2012-11-19, 23:03
+
Patrick Wendell 2012-11-19, 23:04
+
Brock Noland 2012-11-19, 23:29
+
Brock Noland 2012-11-20, 12:25
+
Dan Young 2012-11-20, 15:02
+
Brock Noland 2012-11-20, 16:21
+
Dan Young 2012-11-20, 16:59
+
Brock Noland 2012-11-20, 17:01
+
Dan Young 2012-11-20, 17:10
+
Brock Noland 2012-11-20, 17:14
+
Dan Young 2012-11-20, 17:17
+
Dan Young 2012-11-20, 20:03
+
Brock Noland 2012-11-20, 20:06
Copy link to this message
-
Re: .SpoolingFileLineReader warning....
Ah got it... great! Glad this got figured out.
On Tue, Nov 20, 2012 at 3:06 PM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Great to hear it's working for you!
>
> On Tue, Nov 20, 2012 at 2:03 PM, Dan Young <[EMAIL PROTECTED]> wrote:
> > Just a quick followup on this....I moved all the logging, etc to the same
> > drive/device and I'm no longer seeing those permission errors!.   I'm
> going
> > to keep testing this week.
> >
> > Thank you Brock.
> >
> > Regards,
> >
> > Dano
> >
> >
> > On Tue, Nov 20, 2012 at 10:14 AM, Brock Noland <[EMAIL PROTECTED]>
> wrote:
> >>
> >> OK yeah, so basically it's doing a copy and then a delete. See "man 2
> >> rename" then ERRORS -> EXDEV but basically atomically renaming a file
> >> only works on the same mount point.
> >>
> >> On Tue, Nov 20, 2012 at 11:10 AM, Dan Young <[EMAIL PROTECTED]>
> wrote:
> >> > This is running on an EC2 instance, and these are the ephemeral
> >> > drive(s).
> >> >
> >> > Filesystem     1K-blocks    Used Available Use% Mounted on
> >> > /dev/xvda1       8256952 1618248   6219276  21% /
> >> > udev             1912272       8   1912264   1% /dev
> >> > tmpfs             768100     180    767920   1% /run
> >> > none                5120       0      5120   0% /run/lock
> >> > none             1920244       0   1920244   0% /run/shm
> >> > /dev/xvdb      412814512 1127172 390717564   1% /mnt
> >> >
> >> > ubuntu@ip-foo:~$ cat /etc/fstab
> >> > LABEL=cloudimg-rootfs / ext4 defaults 0 0
> >> > /dev/xvdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2
> >> > ubuntu@ip-foo:~$
> >> >
> >> > ubuntu@ip-foo:~$ mount
> >> > /dev/xvda1 on / type ext4 (rw)
> >> > proc on /proc type proc (rw,noexec,nosuid,nodev)
> >> > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> >> > none on /sys/fs/fuse/connections type fusectl (rw)
> >> > none on /sys/kernel/debug type debugfs (rw)
> >> > none on /sys/kernel/security type securityfs (rw)
> >> > udev on /dev type devtmpfs (rw,mode=0755)
> >> > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> >> > tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> >> > none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> >> > none on /run/shm type tmpfs (rw,nosuid,nodev)
> >> > /dev/xvdb on /mnt type ext3 (rw)
> >> >
> >> > I do see that / is ext4 and /mnt is ext3.....hmm, not sure it matters.
> >> >
> >> > Dano
> >> >
> >> >
> >> >
> >> > On Tue, Nov 20, 2012 at 10:01 AM, Brock Noland <[EMAIL PROTECTED]>
> >> > wrote:
> >> >>
> >> >> Are /var/log and /mnt/flume on the same file system? mv across file
> >> >> systems is a cp and then delete.
> >> >>
> >> >> On Tue, Nov 20, 2012 at 10:59 AM, Dan Young <[EMAIL PROTECTED]>
> >> >> wrote:
> >> >> > Coolio, thank you Brock.
> >> >> >
> >> >> > Did a quick test with a mv vs cp.....this is one test, repeated a
> few
> >> >> > times....the smaller file (~60M) seemed to work fine w/ a mv, but a
> >> >> > larger
> >> >> > file we're seeing the same behavior....
> >> >> >
> >> >> >
> >> >> > Start with no logs in the SpoolingDirectory:
> >> >> >
> >> >> > ls -lrt /mnt/flume/clickstream
> >> >> >
> >> >> > /mnt/flume/clickstream:
> >> >> > total 0
> >> >> >
> >> >> >
> >> >> >
> >> >> > Review the logs that will be rotated via logrotate.d into the
> >> >> > respective
> >> >> > directory; /mnt/flume/clickstream
> >> >> >
> >> >> > ls -lrt /var/log/clickstream
> >> >> > /var/log/clickstream:
> >> >> > total 64112
> >> >> > -rw-rw-r-- 1 ubuntu ubuntu 65648336 Nov 20 16:05 clickstream.log
> >> >> >
> >> >> >
> >> >> >
> >> >> > Review logrotate config in /etc/logrotate.d. Note here, I changed
> >> >> > from
> >> >> > cp -p
> >> >> > to a mv.....
> >> >> >
> >> >> > /var/log/clickstream/clickstream.log
> >> >> > {
> >> >> >   missingok
> >> >> >   rotate 3
> >> >> >   compress
> >> >> >   delaycompress
> >> >> >   copytruncate
> >> >> >   notifempty
> >> >> >   size 50M
> >> >> >   dateext
> >> >> >   dateformat -%Y-%m-%d-%s
> >> >> >   create 666 ubuntu ubuntu