Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Exception : Log File is null for id


+
Raymond Ng 2012-10-02, 08:19
+
Raymond Ng 2012-10-02, 08:43
+
Brock Noland 2012-10-02, 11:06
+
Brock Noland 2012-10-02, 11:28
+
Brock Noland 2012-10-05, 18:19
Copy link to this message
-
Re: Exception : Log File is null for id
Brock,  

This looks like FLUME-1417. This logs on the jira show when the problem is hit during startup. I actually managed to get the Log Id is null error during runtime when I was testing that issue, if you change to small file size and checkpoint very often.

Thanks,
Hari

--
Hari Shreedharan
On Friday, October 5, 2012 at 11:19 AM, Brock Noland wrote:

> Hi,
>
> Just curious if you got around this or figured out what was going on?
> Makes me a little nervous about a file channel bug.
>
> Brock
>
> On Tue, Oct 2, 2012 at 6:28 AM, Brock Noland <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > Also, if you could send us your full log that would be great. The
> > email list doesn't take attachements so either:
> >
> > 1) post it on pastbin
> > or
> > 2) zip it and mail it to me directly
> >
> > Brock
> >
> > On Tue, Oct 2, 2012 at 6:06 AM, Brock Noland <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > Hi,
> > >
> > > What version of flume? If trunk (1.3.0-SNAPSHOT) what is the last
> > > patch you have?
> > >
> > > Can you how us a ls -la of your data and checkpoint directories?
> > >
> > > Brock
> > >
> > > On Tue, Oct 2, 2012 at 3:43 AM, Raymond Ng <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > > Just to add more info to this, I've checked the File channel where a
> > > > "ChannelException: Cannot acquire capacity" is reported against, and can see
> > > > file log-1 has the size of 0 and log-2 has over 300 MB of data, comparing
> > > > with another File channel which has files log-2 and log-3 both with data in
> > > > it but no file log-1 is found.
> > > >
> > > > sounds like log-1 is the one causing the "NullPointerException: LogFile is
> > > > null for id 1" below, and when I restarted flume, I get the following
> > > > warning. I can confirm there was no manual tampering in the file channel
> > > > directory
> > > >
> > > > 2012-10-02 09:38:10,231 INFO [conf-file-poller-0]
> > > > DefaultLogicalNodeManager.java - Starting Channel probeFileChannel1
> > > > 2012-10-02 09:38:10,239 INFO [conf-file-poller-0]
> > > > DefaultLogicalNodeManager.java - Starting Channel probeFileChannel3
> > > > 2012-10-02 09:38:10,313 WARN [lifecycleSupervisor-1-2] ReplayHandler.java -
> > > > Hit EOF on /home/user/flume-ng/filechannel3/data/log-1
> > > > 2012-10-02 09:38:10,314 INFO [lifecycleSupervisor-1-1]
> > > > DirectMemoryUtils.java - Unable to get maxDirectMemory from VM:
> > > > NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
> > > > 2012-10-02 09:38:10,317 INFO [lifecycleSupervisor-1-1]
> > > > DirectMemoryUtils.java - Direct Memory Allocation: Allocation = 1048576,
> > > > Allocated = 0, MaxDirectMemorySize = 954466304, Remaining = 954466304
> > > > 2012-10-02 09:38:10,381 WARN [lifecycleSupervisor-1-1] LogFile.java -
> > > > Checkpoint for file(/home/user/flume-ng/filechannel1/data/log-2) is:
> > > > 1349166469095, which is beyond the requested checkpoint time: 0.
> > > > 2012-10-02 09:38:10,381 WARN [lifecycleSupervisor-1-2] LogFile.java -
> > > > Checkpoint for file(/home/user/flume-ng/filechannel3/data/log-2) is:
> > > > 1349166991594, which is beyond the requested checkpoint time: 0.
> > > > 2012-10-02 09:41:52,144 ERROR [lifecycleSupervisor-1-2] ReplayHandler.java -
> > > > Pending takes 32103 exist after the end of replay. Duplicate messages will
> > > > exist in destination.
> > > > 2012-10-02 09:41:52,709 INFO [lifecycleSupervisor-1-2]
> > > > MonitoredCounterGroup.java - Component type: CHANNEL, name:
> > > > probeFileChannel3 started
> > > > 2012-10-02 09:42:31,413 WARN [lifecycleSupervisor-1-1] LogFile.java -
> > > > Checkpoint for file(/home/cluster_admin/flume-ng/filechannel1/data/log-3)
> > > > is: 1349166981020, which is beyond the requested checkpoint time: 0.
> > > > 2012-10-02 09:45:14,836 ERROR [lifecycleSupervisor-1-1] ReplayHandler.java -
> > > > Pending takes 8409 exist after the end of replay. Duplicate messages will
> > > > exist in destination.
> > > > 2012-10-02 09:45:15,453 INFO [lifecycleSupervisor-1-1]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB