Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - .SpoolingFileLineReader warning....


+
Dan Young 2012-11-17, 15:02
+
Brock Noland 2012-11-17, 15:57
+
Dan Young 2012-11-17, 16:01
+
Brock Noland 2012-11-17, 16:15
+
Dan Young 2012-11-17, 16:33
+
Dan Young 2012-11-19, 19:33
+
Patrick Wendell 2012-11-19, 23:03
+
Patrick Wendell 2012-11-19, 23:04
+
Brock Noland 2012-11-19, 23:29
+
Brock Noland 2012-11-20, 12:25
+
Dan Young 2012-11-20, 15:02
+
Brock Noland 2012-11-20, 16:21
+
Dan Young 2012-11-20, 16:59
+
Brock Noland 2012-11-20, 17:01
Copy link to this message
-
Re: .SpoolingFileLineReader warning....
Dan Young 2012-11-20, 17:10
This is running on an EC2 instance, and these are the ephemeral drive(s).

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8256952 1618248   6219276  21% /
udev             1912272       8   1912264   1% /dev
tmpfs             768100     180    767920   1% /run
none                5120       0      5120   0% /run/lock
none             1920244       0   1920244   0% /run/shm
/dev/xvdb      412814512 1127172 390717564   1% /mnt

ubuntu@ip-foo:~$ cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
/dev/xvdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2
ubuntu@ip-foo:~$

ubuntu@ip-foo:~$ mount
/dev/xvda1 on / type ext4 (rw)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/xvdb on /mnt type ext3 (rw)

I do see that / is ext4 and /mnt is ext3.....hmm, not sure it matters.

Dano

On Tue, Nov 20, 2012 at 10:01 AM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Are /var/log and /mnt/flume on the same file system? mv across file
> systems is a cp and then delete.
>
> On Tue, Nov 20, 2012 at 10:59 AM, Dan Young <[EMAIL PROTECTED]> wrote:
> > Coolio, thank you Brock.
> >
> > Did a quick test with a mv vs cp.....this is one test, repeated a few
> > times....the smaller file (~60M) seemed to work fine w/ a mv, but a
> larger
> > file we're seeing the same behavior....
> >
> >
> > Start with no logs in the SpoolingDirectory:
> >
> > ls -lrt /mnt/flume/clickstream
> >
> > /mnt/flume/clickstream:
> > total 0
> >
> >
> >
> > Review the logs that will be rotated via logrotate.d into the respective
> > directory; /mnt/flume/clickstream
> >
> > ls -lrt /var/log/clickstream
> > /var/log/clickstream:
> > total 64112
> > -rw-rw-r-- 1 ubuntu ubuntu 65648336 Nov 20 16:05 clickstream.log
> >
> >
> >
> > Review logrotate config in /etc/logrotate.d. Note here, I changed from
> cp -p
> > to a mv.....
> >
> > /var/log/clickstream/clickstream.log
> > {
> >   missingok
> >   rotate 3
> >   compress
> >   delaycompress
> >   copytruncate
> >   notifempty
> >   size 50M
> >   dateext
> >   dateformat -%Y-%m-%d-%s
> >   create 666 ubuntu ubuntu
> >   postrotate
> >   mv $1 /mnt/flume/clickstream/ 2>&1
> >   endscript
> > }
> >
> >
> > I run logrotate.d/clickstream.POST with the ~60MB file, and everything
> > looked fine....now I try a ~190MB file....
> >
> >
> >
> > ls -lrt /var/log/clickstream
> > /var/log/clickstream:
> > total 192336
> > -rw-rw-r-- 1 ubuntu ubuntu 196945008 Nov 20 16:42 clickstream.log
> >
> >
> > Run logrotate.d/clickstream.POST, and we see the WARNING in the FLume
> log.
> >
> > ....
> > ....
> > 20 Nov 2012 16:45:07,117 WARN  [pool-13-thread-1]
> > (org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile:328)  -
> > Could not find file:
> > /mnt/flume/clickstream/clickstream.log-2012-11-20-1353429906
> > java.io.FileNotFoundException:
> > /mnt/flume/clickstream/clickstream.log-2012-11-20-1353429906 (Permission
> > denied)
> > at java.io.FileInputStream.open(Native Method)
> > at java.io.FileInputStream.<init>(FileInputStream.java:138)
> > at java.io.FileReader.<init>(FileReader.java:72)
> > at
> >
> org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
> > at
> >
> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
> > at
> >
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
+
Brock Noland 2012-11-20, 17:14
+
Dan Young 2012-11-20, 17:17
+
Dan Young 2012-11-20, 20:03
+
Brock Noland 2012-11-20, 20:06
+
Patrick Wendell 2012-11-23, 12:46