Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: Maven Cloudera Configuration problem


+
Raj K Singh 2013-08-14, 09:05
+
Pavan Sudheendra 2013-08-13, 16:07
+
sandy.ryza@... 2013-08-13, 17:03
+
Pavan Sudheendra 2013-08-13, 17:07
+
Shahab Yunus 2013-08-13, 20:04
+
Brad Cox 2013-08-13, 22:07
+
Suresh Srinivas 2013-08-13, 23:54
Copy link to this message
-
Re: Maven Cloudera Configuration problem
In our Clouder 4.2.0 cluster, Iog-in with *admin* user (do you have
appropriate permissions by the way?) Then I click on any one of the 3
services (hbase, mapred, hdfs and excluding zookeeper) from the top-leftish
menu. Then for each of these I can click the *Configuration* tab which is
in the top-middlish section of the page. Once the configuration page opens
then I click on Action menu on the top-right. One of the sub-menu of this
is *Download Client Configuration* which as the name says downloads the
config files (zip file to be exact) to be used at client machines.

Regards,
Shahab
On Tue, Aug 13, 2013 at 6:07 PM, Brad Cox <[EMAIL PROTECTED]> wrote:

> That link got my hopes up. But Cloudera Manager  (what I'm running; on
> CDH4) does not offer an "Export Client Config" option. What am I missing?
>
> On Aug 13, 2013, at 4:04 PM, Shahab Yunus <[EMAIL PROTECTED]> wrote:
>
> You should not use LocalJobRunner. Make sure that the mapred.job.tracker
> property does not point to 'local' an instead to your job-tracker host and
> port.
>
> *But before that* as Sandy said, your client machine (from where you will
> be kicking of your jobs and apps) should be using config files which will
> have your cluster's configuration. This is the alternative that you should
> follow if you don't want to bundle the configs for your cluster in the
> application itself (either in java code or separate copies of relevant
> properties set of config files.) This was something which I was suggesting
> early on to just to get you started using your cluster instead of local
> mode.
>
> By the way have you seen the following link? It gives you step by step
> information about how to generate config files from your cluster specific
> to your cluster and then how to place them and use the from any machine you
> want to designate as your client. Running your jobs form one of the
> datanodes without proper config would not work.
>
> https://ccp.cloudera.com/display/FREE373/Generating+Client+Configuration
>
> Regards,
> Shahab
>
>
> On Tue, Aug 13, 2013 at 1:07 PM, Pavan Sudheendra <[EMAIL PROTECTED]
> >wrote:
>
> Yes Sandy, I'm referring to LocalJobRunner. I'm actually running the
> job on one datanode..
>
> What changes should i make so that my application would take advantage
> of the cluster as a whole?
>
> On Tue, Aug 13, 2013 at 10:33 PM,  <[EMAIL PROTECTED]> wrote:
>
> Nothing in your pom.xml should affect the configurations your job runs
>
> with.
>
>
> Are you running your job from a node on the cluster? When you say
>
> localhost configurations, do you mean it's using the LocalJobRunner?
>
>
> -sandy
>
> (iphnoe tpying)
>
> On Aug 13, 2013, at 9:07 AM, Pavan Sudheendra <[EMAIL PROTECTED]>
>
> wrote:
>
>
> When i actually run the job on the multi node cluster, logs shows it
> uses localhost configurations which i don't want..
>
> I just have a pom.xml which lists all the dependencies like standard
> hadoop, standard hbase, standard zookeeper etc., Should i remove these
> dependencies?
>
> I want the cluster settings to apply in my map-reduce application..
> So, this is where i'm stuck at..
>
> On Tue, Aug 13, 2013 at 9:30 PM, Pavan Sudheendra <[EMAIL PROTECTED]>
>
> wrote:
>
> Hi Shabab and Sandy,
> The thing is we have a 6 node cloudera cluster running.. For
> development purposes, i was building a map-reduce application on a
> single node apache distribution hadoop with maven..
>
> To be frank, i don't know how to deploy this application on a multi
> node cloudera cluster. I am fairly well versed with Multi Node Apache
> Hadoop Distribution.. So, how can i go forward?
>
> Thanks for all the help :)
>
> On Tue, Aug 13, 2013 at 9:22 PM,  <[EMAIL PROTECTED]> wrote:
>
> Hi Pavan,
>
> Configuration properties generally aren't included in the jar itself
>
> unless you explicitly set them in your java code. Rather they're picked up
> from the mapred-site.xml file located in the Hadoop configuration directory
> on the host you're running your job from.
+
Brad Cox 2013-08-13, 16:34