|
|
-
0.90.4 WrongRegionException
Vladimir Rodionov 2011-12-30, 19:50
This is stack trace of exception we got during routine flow execution. Flow failed. [2011-12-30 12:32:16] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] (HConnectionManager.java:1123) - Retry 19, sleep for 256000ms! INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] (HConnectionManager.java:1123) - Retry 19, sleep for 256000ms! INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - ERROR [2011-12-30 12:36:32] (FinalizeOutputStage.java:142) - execute: Stage FinalizeOutputStage FAILED with exception: Failed 19 actions: WrongRegionException: 19 times, servers with issues: us01-ciqps1-grid03.carrieriq.com:60020, INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 19 actions: WrongRegionException: 19 times, servers with issues: us01-ciqps1-grid03.carrieriq.com:60020, INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1239) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1253) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:836) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.platform.mmp2.input.StripedHBaseTable.flushTable(StripedHBaseTable.java:426) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.platform.mmp2.input.StripedHBaseTable.flush(StripedHBaseTable.java:452) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.platform.mmp3.output.hbase.FactsToHBaseJob.commitHBaseTables(FactsToHBaseJob.java:327) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.platform.stages.OutputStage.commitJobs(OutputStage.java:681) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.platform.stages.FinalizeOutputStage.doExecute(FinalizeOutputStage.java:140) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.platform.stages.OutputStage.execute(OutputStage.java:745) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.platform.flows.FlowExecutor.submitWorkflow(FlowExecutor.java:226) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.platform.flows.FlowExecutor.submitWorkflow(FlowExecutor.java:464) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.flows.FlowExecutionTool.start(FlowExecutionTool.java:239) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.flows.FlowExecutionTool.exec(FlowExecutionTool.java:169) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.flows.FlowExecutionTool.oneRun(FlowExecutionTool.java:545) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.flows.FlowExecutionTool.mainRunResult(FlowExecutionTool.java:580) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.flows.FlowExecutionTool.mainRun(FlowExecutionTool.java:592) INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at com.carrieriq.m2m.flows.FlowExecutionTool.
+
Vladimir Rodionov 2011-12-30, 19:50
-
Re: 0.90.4 WrongRegionException
yuzhihong@... 2011-12-30, 20:25
Can you try 0.90.5 release which has offline Meta builder ?
Cheers
On Dec 30, 2011, at 11:50 AM, Vladimir Rodionov <[EMAIL PROTECTED]> wrote:
> This is stack trace of exception we got during routine flow execution. Flow failed. > > > [2011-12-30 12:32:16] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows > INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows > INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows > INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] (HConnectionManager.java:1123) - Retry 19, sleep for 256000ms! > INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] (HConnectionManager.java:1123) - Retry 19, sleep for 256000ms! > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows > > -- skipped -- > > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] (MetaScanner.java:166) - Scanning .META. starting at row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - ERROR [2011-12-30 12:36:32] (FinalizeOutputStage.java:142) - execute: Stage FinalizeOutputStage FAILED with exception: Failed 19 actions: WrongRegionException: 19 times, servers with issues: us01-ciqps1-grid03.carrieriq.com:60020, > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 19 actions: WrongRegionException: 19 times, servers with issues: us01-ciqps1-grid03.carrieriq.com:60020, > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1239) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1253)
+
yuzhihong@... 2011-12-30, 20:25
|
|