Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - hbase map reduce is talking lot of time


Copy link to this message
-
Re: hbase map reduce is talking lot of time
shashwat shriparv 2012-07-17, 18:48
Hi Syed,

The problem is with the disk space. as map-reduce keeps the intermediate
result on the local disk, just check if you have enough disk space. and
also make sure that you have cleared the tmp directory and its writable.
Just provide more space and try else try with small number of users and
check if its working

Regards


Shashwat Shriparv

On Tue, Jul 17, 2012 at 11:50 AM, syed kather <[EMAIL PROTECTED]> wrote:

> Team ,
>      i had wrote a mapreduce program . scenario of my program is to emit
> <userid,seqid>  .
>
>    Total no user : 825
>    Total no seqid:6583100
>
>    No of map which the program will emit is : 825 * 6583100
>
>   I have Hbase table called ObjectSequence : which consist of 6583100(rows)
>
> i had use TableMapper and TableReducer for my map reduce program
>
>
>  Problem definition :
>
> Processor : i7
> Replication Factor : 1
> Live Datanodes : 3
>
>  Node Last
> Contact  Admin State Configured
> Capacity (GB)  Used
> (GB) Non DFS
> Used (GB) Remaining
> (GB)  Used
> (%) Used
> (%)  Remaining
> (%) Blocks chethan 1In Service 28.590.625.172.82 2.11
>
> 9.8773 shashwat<
> http://shashwat:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
> 2In Service28.980.87 22.016.13
>
> 21.0469 syed<
> http://syed:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
> 0In Service28.984.29 18.376.3214.8
>
> 21.82129
> When i run balancer in hadoop i had seen Blocks are not equally distributed
> . Can i know what may be the reason for this ..
>
>
> Kind% CompleteNum TasksPendingRunningComplete KilledFailed/Killed
> Task Attempts<
> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007>
> map<
> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1
> >
> 85.71%
>
> 701<
> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1&state=running
> >
> 6<
> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1&state=completed
> >
> 0 3<
> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007&kind=map&cause=failed
> >/
> 1<
> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007&kind=map&cause=killed
> >
> reduce<
> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=reduce&pagenum=1
> >
> 28.57%
>
> 101<
> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=reduce&pagenum=1&state=running
> >
> 000 / 0
> i had seen only Number Task is allocated is 8 . Is there any possibility to
> increase the Map Number of Task
>
> Completed TasksTaskCompleteStatusStart TimeFinish TimeErrorsCounters
> task_201207121836_0007_m_000001<
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000001
> >
> 100.00%
> UserID: 777 SEQID:415794
> 12-Jul-2012 21:35:48
> 12-Jul-2012 21:36:12 (24sec)
>
> 16<
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000001>
> task_201207121836_0007_m_000002<
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000002
> >
> 100.00%
> UserID: 777 SEQID:422256
> 12-Jul-2012 21:35:50
> 12-Jul-2012 21:36:47 (57sec)
>
> 16<
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000002>
> task_201207121836_0007_m_000003<
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000003
> >
> 100.00%
> UserID: 777 SEQID:563544
> 12-Jul-2012 21:35:50
> 12-Jul-2012 22:00:08 (24mins, 17sec)
>
> 16<
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000003>
> task_201207121836_0007_m_000004<
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000004
> >
> 100.00%
> UserID: 777 SEQID:592918
> 12-Jul-2012 21:35:50
> 12-Jul-2012 21:42:09 (6mins, 18sec)
>
> 16<
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000004>
> task_201207121836_0007_m_000005<
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000005
> >
> 100.00%
> UserID: 777 SEQID:618121
> 12-Jul-2012 21:35:50
> 12-Jul-2012 21:44:34 (8mins, 43sec)
>
> 16<
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000005>


Shashwat Shriparv