Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> hbase map reduce is talking lot of time


Copy link to this message
-
Re: hbase map reduce is talking lot of time
Hi Syed,

Do give try to this solution :

http://agiletesting.blogspot.in/2011/11/troubleshooting-memory-allocation.html

Regards


Shashwat Shriparv

On Wed, Jul 18, 2012 at 12:18 AM, shashwat shriparv <
[EMAIL PROTECTED]> wrote:

> Hi Syed,
>
> The problem is with the disk space. as map-reduce keeps the intermediate
> result on the local disk, just check if you have enough disk space. and
> also make sure that you have cleared the tmp directory and its writable.
> Just provide more space and try else try with small number of users and
> check if its working
>
> Regards
>
> ∞
> Shashwat Shriparv
>
>
>
>
> On Tue, Jul 17, 2012 at 11:50 AM, syed kather <[EMAIL PROTECTED]> wrote:
>
>> Team ,
>>      i had wrote a mapreduce program . scenario of my program is to emit
>> <userid,seqid>  .
>>
>>    Total no user : 825
>>    Total no seqid:6583100
>>
>>    No of map which the program will emit is : 825 * 6583100
>>
>>   I have Hbase table called ObjectSequence : which consist of
>> 6583100(rows)
>>
>> i had use TableMapper and TableReducer for my map reduce program
>>
>>
>>  Problem definition :
>>
>> Processor : i7
>> Replication Factor : 1
>> Live Datanodes : 3
>>
>>  Node Last
>> Contact  Admin State Configured
>> Capacity (GB)  Used
>> (GB) Non DFS
>> Used (GB) Remaining
>> (GB)  Used
>> (%) Used
>> (%)  Remaining
>> (%) Blocks chethan 1In Service 28.590.625.172.82 2.11
>>
>> 9.8773 shashwat<
>> http://shashwat:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
>> 2In Service28.980.87 22.016.13
>>
>> 21.0469 syed<
>> http://syed:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
>> 0In Service28.984.29 18.376.3214.8
>>
>> 21.82129
>> When i run balancer in hadoop i had seen Blocks are not equally
>> distributed
>> . Can i know what may be the reason for this ..
>>
>>
>> Kind% CompleteNum TasksPendingRunningComplete KilledFailed/Killed
>> Task Attempts<
>> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007>
>> map<
>> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1
>> >
>> 85.71%
>>
>> 701<
>> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1&state=running
>> >
>> 6<
>> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1&state=completed
>> >
>> 0 3<
>> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007&kind=map&cause=failed
>> >/
>> 1<
>> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007&kind=map&cause=killed
>> >
>> reduce<
>> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=reduce&pagenum=1
>> >
>> 28.57%
>>
>> 101<
>> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=reduce&pagenum=1&state=running
>> >
>> 000 / 0
>> i had seen only Number Task is allocated is 8 . Is there any possibility
>> to
>> increase the Map Number of Task
>>
>> Completed TasksTaskCompleteStatusStart TimeFinish TimeErrorsCounters
>> task_201207121836_0007_m_000001<
>> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000001
>> >
>> 100.00%
>> UserID: 777 SEQID:415794
>> 12-Jul-2012 21:35:48
>> 12-Jul-2012 21:36:12 (24sec)
>>
>> 16<
>> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000001>
>> task_201207121836_0007_m_000002<
>> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000002
>> >
>> 100.00%
>> UserID: 777 SEQID:422256
>> 12-Jul-2012 21:35:50
>> 12-Jul-2012 21:36:47 (57sec)
>>
>> 16<
>> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000002>
>> task_201207121836_0007_m_000003<
>> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000003
>> >
>> 100.00%
>> UserID: 777 SEQID:563544
>> 12-Jul-2012 21:35:50
>> 12-Jul-2012 22:00:08 (24mins, 17sec)
>>
>> 16<
>> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000003>
>> task_201207121836_0007_m_000004<
>> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000004
>> >
>> 100.00%
>> UserID: 777 SEQID:592918

Shashwat Shriparv
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB