Hive, mail # user - Partition column on an Alpha Numeric Column - 2014-02-11, 17:18
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Partition column on an Alpha Numeric Column
All,

One of the primary key columns in a Relational table has alpha numberic of 6 characters - varchar(6).

The first three characters has this pattern -
1st one - 1 to 9   

2nd one - 1 to 9 or a -z
3rd one - 1 to 9 or a -z

Is this a good idea for performing queries ( can be any queries based on other columns of the table )

Partition the data based on the first three characters summing upto a total of 10 * 36 * 36 which is 12,960 partitons.
12960 partitons - Is it too much ? Impossible or never heard ? or can we consider this design ?

I know NameNode should have a powerful RAM. But how much ? How do we determine the limitation of the number of files a Name Node can handle?

Thanks,
Raj
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB