Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Disk space usage of HFilev1 vs HFilev2


Copy link to this message
-
Re: Disk space usage of HFilev1 vs HFilev2
Hi Kevin,

If it's not possible to store table in HFilev1 in HBase 0.92 then my last
option will be to do store data on pseudo-distributed or standalone cluster
for the comparison.
The advantage with the current installation is that its a fully distributed
cluster with around 33 million records in a table. So, it would give me a
better estimate.

Thanks,
Anil Gupta

On Tue, Aug 14, 2012 at 1:48 PM, Kevin O'dell <[EMAIL PROTECTED]>wrote:

> Do you not have a pseudo cluster for testing anywhere?
>
> On Tue, Aug 14, 2012 at 4:46 PM, anil gupta <[EMAIL PROTECTED]> wrote:
>
> > Hi Jerry,
> >
> > I am wiling to do that but the problem is that i wiped off the HBase0.90
> > cluster. Is there a way to store a table in HFilev1 in HBase0.92? If i
> can
> > store a file in HFilev1 in 0.92 then i can do the comparison.
> >
> > Thanks,
> > Anil Gupta
> >
> > On Tue, Aug 14, 2012 at 1:28 PM, Jerry Lam <[EMAIL PROTECTED]> wrote:
> >
> > > Hi Anil:
> > >
> > > Maybe you can try to compare the two HFile implementation directly? Let
> > say
> > > write 1000 rows into HFile v1 format and then into HFile v2 format. You
> > can
> > > then compare the size of the two directly?
> > >
> > > HTH,
> > >
> > > Jerry
> > >
> > > On Tue, Aug 14, 2012 at 3:36 PM, anil gupta <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi Zahoor,
> > > >
> > > > Then it seems like i might have missed something when doing hdfs
> usage
> > > > estimation of HBase. I usually do hadoop fs -dus /hbase/$TABLE_NAME
> for
> > > > getting the hdfs usage of a table. Is this the right way? Since i
> wiped
> > > of
> > > > the HBase0.90 cluster so now i cannot look into hdfs usage of it. Is
> it
> > > > possible to store a table in HFileV1 instead of HFileV2 in HBase0.92?
> > > > In this way i can do a fair comparison.
> > > >
> > > > Thanks,
> > > > Anil Gupta
> > > >
> > > > On Tue, Aug 14, 2012 at 12:13 PM, jmozah <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Hi Anil,
> > > > >
> > > > > I really doubt that there is 50% drop in file sizes... As far as i
> > > know..
> > > > > there is no drastic space conserving feature in V2. Just as  an
> after
> > > > > thought.. do a major compact and check the sizes.
> > > > >
> > > > > ./Zahoor
> > > > > http://blog.zahoor.in
> > > > >
> > > > >
> > > > > On 15-Aug-2012, at 12:31 AM, anil gupta <[EMAIL PROTECTED]>
> > wrote:
> > > > >
> > > > > > l
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Thanks & Regards,
> > > > Anil Gupta
> > > >
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

--
Thanks & Regards,
Anil Gupta
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB