Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Disk space usage of HFilev1 vs HFilev2


+
anil gupta 2012-08-14, 18:15
+
jmozah 2012-08-14, 18:50
+
anil gupta 2012-08-14, 19:01
+
jmozah 2012-08-14, 19:13
+
anil gupta 2012-08-14, 19:36
+
Jerry Lam 2012-08-14, 20:28
+
anil gupta 2012-08-14, 20:46
+
Kevin Odell 2012-08-14, 20:48
Copy link to this message
-
Re: Disk space usage of HFilev1 vs HFilev2
anil gupta 2012-08-14, 20:55
Hi Kevin,

If it's not possible to store table in HFilev1 in HBase 0.92 then my last
option will be to do store data on pseudo-distributed or standalone cluster
for the comparison.
The advantage with the current installation is that its a fully distributed
cluster with around 33 million records in a table. So, it would give me a
better estimate.

Thanks,
Anil Gupta

On Tue, Aug 14, 2012 at 1:48 PM, Kevin O'dell <[EMAIL PROTECTED]>wrote:

> Do you not have a pseudo cluster for testing anywhere?
>
> On Tue, Aug 14, 2012 at 4:46 PM, anil gupta <[EMAIL PROTECTED]> wrote:
>
> > Hi Jerry,
> >
> > I am wiling to do that but the problem is that i wiped off the HBase0.90
> > cluster. Is there a way to store a table in HFilev1 in HBase0.92? If i
> can
> > store a file in HFilev1 in 0.92 then i can do the comparison.
> >
> > Thanks,
> > Anil Gupta
> >
> > On Tue, Aug 14, 2012 at 1:28 PM, Jerry Lam <[EMAIL PROTECTED]> wrote:
> >
> > > Hi Anil:
> > >
> > > Maybe you can try to compare the two HFile implementation directly? Let
> > say
> > > write 1000 rows into HFile v1 format and then into HFile v2 format. You
> > can
> > > then compare the size of the two directly?
> > >
> > > HTH,
> > >
> > > Jerry
> > >
> > > On Tue, Aug 14, 2012 at 3:36 PM, anil gupta <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi Zahoor,
> > > >
> > > > Then it seems like i might have missed something when doing hdfs
> usage
> > > > estimation of HBase. I usually do hadoop fs -dus /hbase/$TABLE_NAME
> for
> > > > getting the hdfs usage of a table. Is this the right way? Since i
> wiped
> > > of
> > > > the HBase0.90 cluster so now i cannot look into hdfs usage of it. Is
> it
> > > > possible to store a table in HFileV1 instead of HFileV2 in HBase0.92?
> > > > In this way i can do a fair comparison.
> > > >
> > > > Thanks,
> > > > Anil Gupta
> > > >
> > > > On Tue, Aug 14, 2012 at 12:13 PM, jmozah <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Hi Anil,
> > > > >
> > > > > I really doubt that there is 50% drop in file sizes... As far as i
> > > know..
> > > > > there is no drastic space conserving feature in V2. Just as  an
> after
> > > > > thought.. do a major compact and check the sizes.
> > > > >
> > > > > ./Zahoor
> > > > > http://blog.zahoor.in
> > > > >
> > > > >
> > > > > On 15-Aug-2012, at 12:31 AM, anil gupta <[EMAIL PROTECTED]>
> > wrote:
> > > > >
> > > > > > l
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Thanks & Regards,
> > > > Anil Gupta
> > > >
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

--
Thanks & Regards,
Anil Gupta
+
Harsh J 2012-08-14, 21:42
+
J Mohamed Zahoor 2012-08-15, 12:09
+
anil gupta 2012-08-27, 20:19
+
Kevin Odell 2012-08-27, 20:32
+
anil gupta 2012-08-28, 03:30
+
Stack 2012-08-28, 14:37
+
Matt Corgan 2012-08-28, 18:24
+
lars hofhansl 2012-08-28, 18:42
+
Stack 2012-08-28, 18:54
+
lars hofhansl 2012-08-28, 19:08
+
anil gupta 2012-08-14, 21:52