Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Re: High Full GC count for Region server


+
Jean-Marc Spaggiari 2013-10-22, 14:45
+
Vimal Jain 2013-10-22, 16:01
+
Ted Yu 2013-10-22, 16:49
+
Jean-Marc Spaggiari 2013-10-24, 13:58
Copy link to this message
-
Re: High Full GC count for Region server
Asaf Mesika 2013-10-28, 18:25
Check through HDFS UI that your cluster haven't reached maximum disk
capacity

On Thursday, October 24, 2013, Vimal Jain wrote:

> Hi Ted/Jean,
> Can you please help here ?
>
>
> On Tue, Oct 22, 2013 at 10:29 PM, Vimal Jain <[EMAIL PROTECTED]<javascript:;>>
> wrote:
>
> > Hi Ted,
> > Yes i checked namenode and datanode logs and i found below exceptions in
> > both the logs:-
> >
> > Name node :-
> > java.io.IOException: File
> >
> /hbase/event_data/433b61f2a4ebff8f2e4b89890508a3b7/.tmp/99797a61a8f7471cb6df8f7b95f18e9e
> > could only be replicated to 0 nodes, instead of 1
> >
> > java.io.IOException: Got blockReceived message from unregistered or dead
> > node blk_-2949905629769882833_52274
> >
> > Data node :-
> > 480000 millis timeout while waiting for channel to be ready for write. ch
> > : java.nio.channels.SocketChannel[connected local=/192.168.20.30:50010
> >  remote=/192.168.20.30:36188]
> >
> > ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> > DatanodeRegistration(192.168.20.30:50010,
> > storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
> infoPort=50075,
> > ipcPort=50020):DataXceiver
> >
> > java.io.EOFException: while trying to read 39309 bytes
> >
> >
> > On Tue, Oct 22, 2013 at 10:19 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> >> bq. java.io.IOException: File /hbase/event_data/
> >> 4c3765c51911d6c67037a983d205a010/.tmp/bfaf8df33d5b4068825e3664d3e4b2b0
> >> could
> >> only be replicated to 0 nodes, instead of 1
> >>
> >> Have you checked Namenode / Datanode logs ?
> >> Looks like hdfs was not stable.
> >>
> >>
> >> On Tue, Oct 22, 2013 at 9:01 AM, Vimal Jain <[EMAIL PROTECTED]> wrote:
> >>
> >> > HI Jean,
> >> > Thanks for your reply.
> >> > I have total 8 GB memory and distribution is as follows:-
> >> >
> >> > Region server  - 2 GB
> >> > Master,Namenode,Datanode,Secondary Namenode,Zookepeer - 1 GB
> >> > OS - 1 GB
> >> >
> >> > Please let me know if you need more information.
> >> >
> >> >
> >> > On Tue, Oct 22, 2013 at 8:15 PM, Jean-Marc Spaggiari <
> >> > [EMAIL PROTECTED]> wrote:
> >> >
> >> > > Hi Vimal,
> >> > >
> >> > > What are your settings? Memory of the host, and memory allocated for
> >> the
> >> > > different HBase services?
> >> > >
> >> > > Thanks,
> >> > >
> >> > > JM
> >> > >
> >> > >
> >> > > 2013/10/22 Vimal Jain <[EMAIL PROTECTED]>
> >> > >
> >> > > > Hi,
> >> > > > I am running in Hbase in pseudo distributed mode. ( Hadoop
> version -
> >> > > 1.1.2
> >> > > > , Hbase version - 0.94.7 )
> >> > > > I am getting few exceptions in both hadoop ( namenode , datanode)
> >> logs
> >> > > and
> >> > > > hbase(region server).
> >> > > > When i search for these exceptions on google , i concluded  that
> >> > problem
> >> > > is
> >> > > > mainly due to large number of full GC in region server process.
> >> > > >
> >> > > > I used jstat and found that there are total of 950 full GCs in
> span
> >> of
> >> > 4
> >> > > > days for region server process.Is this ok?
> >> > > >
> >> > > > I am totally confused by number of exceptions i am getting.
> >> > > > Also i get below exceptions intermittently.
> >> > > >
> >> > > >
> >> > > > Region server:-
> >> > > >
> >> > > > 2013-10-22 12:00:26,627 WARN org.apache.hadoop.ipc.HBaseServer:
> >> > > > (responseTooSlow):
> >> > > > {"processingtimems":15312,"call":"next(-6681408251916104762,
> 1000),
> >> rpc
> >> > > > version=1, client version=29,
> >> > methodsFingerPrint=-1368823753","client":"
> >> > > > 192.168.20.31:48270
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> ","starttimems":1382423411293,"queuetimems":0,"class":"HRegionServer","responsesize":4808556,"method":"next"}
> >> > > > 2013-10-22 12:06:17,606 WARN org.apache.hadoop.ipc.HBaseServer:
> >> > > > (operationTooSlow): {"processingtimems":14759,"client":"
> >> > > > 192.168.20.31:48247
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> ","timeRange":[0,9223372036854775807],"starttimems":1382423762845,"responsesize":61,"class":"HRegionServer","table":"event_data","cacheBlocks":true,"families":{"gin
+
Vimal Jain 2013-10-31, 08:52
+
Adrien Mogenet 2013-10-31, 10:20