Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> issue about search speed and rowkey


+
ch huang 2013-08-07, 09:18
+
Lu, Wei 2013-08-07, 12:11
Copy link to this message
-
Re: issue about search speed and rowkey
Table is closed for each Put in writeRow(). This is not efficient.

Take a look at http://hbase.apache.org/book.html#client , 9.3.1 Connections

On Wed, Aug 7, 2013 at 5:11 AM, Lu, Wei <[EMAIL PROTECTED]> wrote:

> decrease cache size (say, 1000)  and increase batch or just set it as
> default if #qualifiers in a row is not too many.
>
>
> -----Original Message-----
> From: ch huang [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, August 07, 2013 5:18 PM
> To: [EMAIL PROTECTED]
> Subject: issue about search speed and rowkey
>
> hi,all:
>
>         i have problem in running following code,MoveData method is used to
> get data from source table ,and modify each row rowkey ,and insert into
> destination table,and i always get error,anyone can help?
>
>  public static void writeRow(HTablePool htp,String tablename, String
> rowkey,String cf,String col,String value) {
>         try {
>          HTableInterface table = htp.getTable(Bytes.toBytes(tablename));
>             Put put = new Put(Bytes.toBytes(rowkey));
>                 put.add(Bytes.toBytes(cf),
>                   Bytes.toBytes(col),
>                   Long.parseLong(rowkey),
>                   Bytes.toBytes(value));
>
>                 table.put(put);
>             table.close();
>         } catch (IOException e) {
>             e.printStackTrace();
>         }
>     }
>
> public static void MoveData(String src_t,String dest_t){
>
>     try{
>      HTable tabsrc = new HTable(conf, src_t);
>      HTable tabdest = new HTable(conf, dest_t);
>      tabsrc.setAutoFlush(false);
>      tabdest.setAutoFlush(false);
>      HTablePool tablePool = new HTablePool(conf, 5);
>      Scan scan = new Scan();
>      scan.setCaching(10000);
>         scan.setBatch(10);
>         ResultScanner rs = tabsrc.getScanner(scan);
>
>            for (Result r : rs){
>             ArrayList al = new ArrayList();
>             HashMap hm = new HashMap();
>             for (KeyValue kv : r.raw()){
>
>              hm.put(new String(kv.getQualifier()), new
> String(kv.getValue()));
>              al.add(new String(kv.getQualifier()));
>             }
>
>                for (int i = 0; i < al.size(); i++) {
>
>
> writeRow(tablePool,dest_t,hm.get("date").toString(),"info",al.get(i).toString(),hm.get(al.get(i)).toString());
>                }
>          }
>
>            rs.close();
>            tabsrc.close();
>            tabdest.close();
>
>     }catch(IOException e){
>      e.printStackTrace();
>     }
>
>    }
>
>
>
> 2013-08-07 16:43:31,250 WARN  [main] conf.Configuration
> (Configuration.java:warnOnceIfDeprecated(824)) - hadoop.native.lib is
> deprecated. Instead, use io.native.lib.available
> java.lang.RuntimeException:
> org.apache.hadoop.hbase.client.ScannerTimeoutException: 123891ms passed
> since the last invocation, timeout is currently set to 120000
>  at
>
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:44)
>  at com.testme.demo.HBaseTest.MoveData(HBaseTest.java:186)
>  at com.testme.demo.HBaseTest.main(HBaseTest.java:314)
> Caused by: org.apache.hadoop.hbase.client.ScannerTimeoutException: 123891ms
> passed since the last invocation, timeout is currently set to 120000
>  at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:283)
>  at
>
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:41)
>  ... 2 more
> Caused by: org.apache.hadoop.hbase.UnknownScannerException:
> org.apache.hadoop.hbase.UnknownScannerException: Name: -1792412350530007203
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2464)
>  at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
>  at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
+
ch huang 2013-08-08, 01:08
+
Ted Yu 2013-08-08, 01:15