Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - issue about search speed and rowkey


Copy link to this message
-
Re: issue about search speed and rowkey
ch huang 2013-08-08, 01:08
table.close(); this is not close the table ,just get the connect back to
pool,because

*putTable methode in HTablePool is Deprecated,see*

http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTablePool.html#putTable(org.apache.hadoop.hbase.client.HTableInterface)
On Wed, Aug 7, 2013 at 9:46 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Table is closed for each Put in writeRow(). This is not efficient.
>
> Take a look at http://hbase.apache.org/book.html#client , 9.3.1
> Connections
>
> On Wed, Aug 7, 2013 at 5:11 AM, Lu, Wei <[EMAIL PROTECTED]> wrote:
>
> > decrease cache size (say, 1000)  and increase batch or just set it as
> > default if #qualifiers in a row is not too many.
> >
> >
> > -----Original Message-----
> > From: ch huang [mailto:[EMAIL PROTECTED]]
> > Sent: Wednesday, August 07, 2013 5:18 PM
> > To: [EMAIL PROTECTED]
> > Subject: issue about search speed and rowkey
> >
> > hi,all:
> >
> >         i have problem in running following code,MoveData method is used
> to
> > get data from source table ,and modify each row rowkey ,and insert into
> > destination table,and i always get error,anyone can help?
> >
> >  public static void writeRow(HTablePool htp,String tablename, String
> > rowkey,String cf,String col,String value) {
> >         try {
> >          HTableInterface table = htp.getTable(Bytes.toBytes(tablename));
> >             Put put = new Put(Bytes.toBytes(rowkey));
> >                 put.add(Bytes.toBytes(cf),
> >                   Bytes.toBytes(col),
> >                   Long.parseLong(rowkey),
> >                   Bytes.toBytes(value));
> >
> >                 table.put(put);
> >             table.close();
> >         } catch (IOException e) {
> >             e.printStackTrace();
> >         }
> >     }
> >
> > public static void MoveData(String src_t,String dest_t){
> >
> >     try{
> >      HTable tabsrc = new HTable(conf, src_t);
> >      HTable tabdest = new HTable(conf, dest_t);
> >      tabsrc.setAutoFlush(false);
> >      tabdest.setAutoFlush(false);
> >      HTablePool tablePool = new HTablePool(conf, 5);
> >      Scan scan = new Scan();
> >      scan.setCaching(10000);
> >         scan.setBatch(10);
> >         ResultScanner rs = tabsrc.getScanner(scan);
> >
> >            for (Result r : rs){
> >             ArrayList al = new ArrayList();
> >             HashMap hm = new HashMap();
> >             for (KeyValue kv : r.raw()){
> >
> >              hm.put(new String(kv.getQualifier()), new
> > String(kv.getValue()));
> >              al.add(new String(kv.getQualifier()));
> >             }
> >
> >                for (int i = 0; i < al.size(); i++) {
> >
> >
> >
> writeRow(tablePool,dest_t,hm.get("date").toString(),"info",al.get(i).toString(),hm.get(al.get(i)).toString());
> >                }
> >          }
> >
> >            rs.close();
> >            tabsrc.close();
> >            tabdest.close();
> >
> >     }catch(IOException e){
> >      e.printStackTrace();
> >     }
> >
> >    }
> >
> >
> >
> > 2013-08-07 16:43:31,250 WARN  [main] conf.Configuration
> > (Configuration.java:warnOnceIfDeprecated(824)) - hadoop.native.lib is
> > deprecated. Instead, use io.native.lib.available
> > java.lang.RuntimeException:
> > org.apache.hadoop.hbase.client.ScannerTimeoutException: 123891ms passed
> > since the last invocation, timeout is currently set to 120000
> >  at
> >
> >
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:44)
> >  at com.testme.demo.HBaseTest.MoveData(HBaseTest.java:186)
> >  at com.testme.demo.HBaseTest.main(HBaseTest.java:314)
> > Caused by: org.apache.hadoop.hbase.client.ScannerTimeoutException:
> 123891ms
> > passed since the last invocation, timeout is currently set to 120000
> >  at
> > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:283)
> >  at
> >
> >
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:41)
> >  ... 2 more
> > Caused by: org.apache.hadoop.hbase.UnknownScannerException: