-Re: Hadoop Datacenter Setup
Michael Segel 2012-01-31, 02:46
If you are going this route why not net boot the nodes in the cluster?
Sent from my iPhone
On Jan 30, 2012, at 8:17 PM, "Patrick Angeles" <[EMAIL PROTECTED]> wrote:
> Hey Aaron,
> I'm still skeptical when it comes to flash drives, especially as pertains
> to Hadoop. The write cycle limit is impractical to make them usable for
> dfs.data.dir and mapred.local.dir, and as you pointed out, you can't use
> them for logs either.
> If you put HADOOP_LOG_DIR in /mnt/d0, you will still have to shut down the
> TT and DN in order to replace the drive. So you may as well just carve out
> 100GB from that drive and put your root filesystem there.
> I'd say that unless you're running some extremely CPU-heavy workloads, you
> should consider getting more than 3 drives per node. Most shops get 6-12
> drives per node (with dual quad or hex core processors). Then you can
> sacrifice one of the drives for swap and the OS.
> I'd keep the RegionServer heap at 12GB or under to mitigate long GC pauses
> (the bigger the heap, the longer the eventual full GC).
> Finally, you can run Hive on the same cluster as HBase, just be wary of
> load spikes due to MR jobs and configure properly. You don't want a large
> Hive query to knock out your RegionServers thereby causing cascading
> - P
> On Mon, Jan 30, 2012 at 6:44 PM, Aaron Tokhy <
> [EMAIL PROTECTED]> wrote:
>> I forgot to add:
>> Are there use cases for using a swap partition for Hadoop nodes if our
>> combined planned heap size is not expected to go over 24GB for any
>> particular node type? I've noticed that if HBase starts to GC, it will
>> pause for unreasonable amounts of time if old pages get swapped to disk,
>> causing the regionserver to crash (which we've mitigated by setting
>> Our slave node template will have a 1 GB heap Task Tracker, a 1 GB heap
>> Data Node and a 12-16GB heap RegionServer. We assume the OS memory
>> overhead is 1 GB. We added another 1 GB for combined Java VM overhead
>> across services, which comes up to be around a max of 16-20GB used. This
>> gives us around 4-8GB for tasks that would work with HBase. We may also
>> use Hive on the same cluster for queries.
>> On 01/30/2012 05:40 PM, Aaron Tokhy wrote:
>>> Our group is trying to set up a prototype for what will eventually
>>> become a cluster of ~50 nodes.
>>> Anyone have experiences with a stateless Hadoop cluster setup using this
>>> method on CentOS? Are there any caveats with a read-only root file
>>> system approach? This would save us from having to keep a root volume
>>> on every system (whether it is installed on a USB thumb drive, or a RAID
>>> 1 of bootable / partitions).
>>> We would like to keep the OS root file system separate from the Hadoop
>>> filesystem(s) for maintenance reasons (we can hot swap disks while the
>>> system is running)
>>> We were also considering installing the root filesystem on USB flash
>>> drives, making it persistent yet separate. However we would identify
>>> and turn off anything that would cause excess writes to the root
>>> filesystem given the limited number of USB flash drive write cycles
>>> (keep IO writes to the root filesystem to a minimum). We would do this
>>> by storing the Hadoop logs on the same filesystem/drive as what we
>>> specify in dfs.data.dir/dfs.name.dir.
>>> In the end we would have something like this:
>>> USB (MS DOS partition table + 1 ext2/3/4 partition)
>>> /dev/sda1 mounted as / (possibly read-only)
>>> /dev/sda2 mounted as /var (read-write)
>>> /dev/sda3 mounted as /tmp (read-write)
>>> Hadoop Disks (no partition table or GPT since these are 3TB disks)
>>> /dev/sdb /mnt/d0
>>> /dev/sdc /mnt/d1
>>> /dev/sdd /mnt/d2