Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> hbase map reduce is talking lot of time


Copy link to this message
-
Re: hbase map reduce is talking lot of time
Hi Syed,

Do give try to this solution :

http://agiletesting.blogspot.in/2011/11/troubleshooting-memory-allocation.html

Regards


Shashwat Shriparv

On Wed, Jul 18, 2012 at 12:18 AM, shashwat shriparv <
[EMAIL PROTECTED]> wrote:

> Hi Syed,
>
> The problem is with the disk space. as map-reduce keeps the intermediate
> result on the local disk, just check if you have enough disk space. and
> also make sure that you have cleared the tmp directory and its writable.
> Just provide more space and try else try with small number of users and
> check if its working
>
> Regards
>
> ∞
> Shashwat Shriparv
>
>
>
>
> On Tue, Jul 17, 2012 at 11:50 AM, syed kather <[EMAIL PROTECTED]> wrote:
>
>> Team ,
>>      i had wrote a mapreduce program . scenario of my program is to emit
>> <userid,seqid>  .
>>
>>    Total no user : 825
>>    Total no seqid:6583100
>>
>>    No of map which the program will emit is : 825 * 6583100
>>
>>   I have Hbase table called ObjectSequence : which consist of
>> 6583100(rows)
>>
>> i had use TableMapper and TableReducer for my map reduce program
>>
>>
>>  Problem definition :
>>
>> Processor : i7
>> Replication Factor : 1
>> Live Datanodes : 3
>>
>>  Node Last
>> Contact  Admin State Configured
>> Capacity (GB)  Used
>> (GB) Non DFS
>> Used (GB) Remaining
>> (GB)  Used
>> (%) Used
>> (%)  Remaining
>> (%) Blocks chethan 1In Service 28.590.625.172.82 2.11
>>
>> 9.8773 shashwat<
>> http://shashwat:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
>> 2In Service28.980.87 22.016.13
>>
>> 21.0469 syed<
>> http://syed:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
>> 0In Service28.984.29 18.376.3214.8
>>
>> 21.82129
>> When i run balancer in hadoop i had seen Blocks are not equally
>> distributed
>> . Can i know what may be the reason for this ..
>>
>>
>> Kind% CompleteNum TasksPendingRunningComplete KilledFailed/Killed
>> Task Attempts<
>> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007>
>> map<
>> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1
>> >
>> 85.71%
>>
>> 701<
>> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1&state=running
>> >
>> 6<
>> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1&state=completed
>> >
>> 0 3<
>> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007&kind=map&cause=failed
>> >/
>> 1<
>> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007&kind=map&cause=killed
>> >
>> reduce<
>> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=reduce&pagenum=1
>> >
>> 28.57%
>>
>> 101<
>> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=reduce&pagenum=1&state=running
>> >
>> 000 / 0
>> i had seen only Number Task is allocated is 8 . Is there any possibility
>> to
>> increase the Map Number of Task
>>
>> Completed TasksTaskCompleteStatusStart TimeFinish TimeErrorsCounters
>> task_201207121836_0007_m_000001<
>> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000001
>> >
>> 100.00%
>> UserID: 777 SEQID:415794
>> 12-Jul-2012 21:35:48
>> 12-Jul-2012 21:36:12 (24sec)
>>
>> 16<
>> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000001>
>> task_201207121836_0007_m_000002<
>> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000002
>> >
>> 100.00%
>> UserID: 777 SEQID:422256
>> 12-Jul-2012 21:35:50
>> 12-Jul-2012 21:36:47 (57sec)
>>
>> 16<
>> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000002>
>> task_201207121836_0007_m_000003<
>> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000003
>> >
>> 100.00%
>> UserID: 777 SEQID:563544
>> 12-Jul-2012 21:35:50
>> 12-Jul-2012 22:00:08 (24mins, 17sec)
>>
>> 16<
>> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000003>
>> task_201207121836_0007_m_000004<
>> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000004
>> >
>> 100.00%
>> UserID: 777 SEQID:592918

Shashwat Shriparv