Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Partition column on an Alpha Numeric Column


Copy link to this message
-
Partition column on an Alpha Numeric Column
All,

One of the primary key columns in a Relational table has alpha numberic of 6 characters - varchar(6).

The first three characters has this pattern -
1st one - 1 to 9   

2nd one - 1 to 9 or a -z
3rd one - 1 to 9 or a -z

Is this a good idea for performing queries ( can be any queries based on other columns of the table )

Partition the data based on the first three characters summing upto a total of 10 * 36 * 36 which is 12,960 partitons.
12960 partitons - Is it too much ? Impossible or never heard ? or can we consider this design ?

I know NameNode should have a powerful RAM. But how much ? How do we determine the limitation of the number of files a Name Node can handle?

Thanks,
Raj
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB