| clear query|facets|time |
Search criteria: .
Results from 1 to 10 from
81 (2.447s).
|
|
|
Loading phrases to help you
refine your search...
|
|
Re: Poor HBase map-reduce scan performance - HBase - [mail # user]
|
|
...Thanks Enis, I'll see if I can backport this patch - it is exactly what I was going to try. This should solve my scan performance problems if I can get it to work. On May 29, 2013, at ...
|
|
|
Author: Bryan Keller,
2013-06-04, 17:01
|
|
|
Re: Poor HBase map-reduce scan performance - HBase - [mail # user]
|
|
...I am considering scanning a snapshot instead of the table. I believe this is what the ExportSnapshot class does. If I could use the scanning code from ExportSnapshot then I will be able to s...
|
|
|
Author: Bryan Keller,
2013-05-23, 15:45
|
|
|
Re: Poor HBase map-reduce scan performance - HBase - [mail # user]
|
|
...FYI, I ran tests with compression on and off. With a plain HDFS sequence file and compression off, I am getting very good I/O numbers, roughly 75% of theoretical max for reads. With sn...
|
|
|
Author: Bryan Keller,
2013-05-10, 15:46
|
|
|
Re: Poor HBase map-reduce scan performance - HBase - [mail # user]
|
|
...Thanks for the offer Lars! I haven't made much progress speeding things up. I finally put together a test program that populates a table that is similar to my production dataset. I hav...
|
|
|
Author: Bryan Keller,
2013-05-08, 17:15
|
|
|
Re: Poor HBase map-reduce scan performance - HBase - [mail # user]
|
|
...Actually I'm not too confident in my results re block size, they may have been related to major compaction. I'm going to rerun before drawing any conclusions. On May 3, 2013, at 12:17 ...
|
|
|
Author: Bryan Keller,
2013-05-03, 10:44
|
|
|
Re: Poor HBase map-reduce scan performance - HBase - [mail # user]
|
|
...I finally made some progress. I tried a very large HBase block size (16mb), and it significantly improved scan performance. I went from 45-50 min to 24 min. Not great but much better. Before...
|
|
|
Author: Bryan Keller,
2013-05-03, 07:17
|
|
|
Re: Poor HBase map-reduce scan performance - HBase - [mail # user]
|
|
...I ran one of my regionservers through VisualVM. It looks like the top hot spots are HFileReaderV2$ScannerV2.getKeyValue() and ByteBuffer.allocate(). It appears at first glance that memory al...
|
|
|
Author: Bryan Keller,
2013-05-02, 17:54
|
|
|
Re: Poor HBase map-reduce scan performance - HBase - [mail # user]
|
|
...I used exactly 0.94.4, pulled from the tag in subversion. On May 1, 2013, at 9:41 PM, lars hofhansl wrote: ...
|
|
|
Author: Bryan Keller,
2013-05-02, 04:49
|
|
|
Re: Poor HBase map-reduce scan performance - HBase - [mail # user]
|
|
...I tried running my test with 0.94.4, unfortunately performance was about the same. I'm planning on profiling the regionserver and trying some other things tonight and tomorrow and will repor...
|
|
|
Author: Bryan Keller,
2013-05-02, 01:01
|
|
|
Re: Poor HBase map-reduce scan performance - HBase - [mail # user]
|
|
...Yes I have monitored GC, CPU, disk and network IO, anything else I could think of. Only the CPU usage by the regionserver is on the high side. I mentioned data local jobs make up gener...
|
|
|
Author: Bryan Keller,
2013-05-01, 16:39
|
|
|
|
