Hi everyone,
This is my first post in this list, as I am a newb with Hadoop.
I am looking in the web for some documentation and example on how to use DI
framework with Hadoop.

Basically I want to have the mapping / reducing logic outside the mapper /
reducer classes.

I did a small prototype to show POC ofr our project.
In the prototype I had classes with static methods that did the mapping /
reducing logic.
this is not my preferable way of course.

So,
Are there any recommendations for good DI in Hadoop documentation?

Thanks,

Eyal
Eyal Golan
[EMAIL PROTECTED]

Visit: http://jvdrums.sourceforge.net/
LinkedIn: http://www.linkedin.com/in/egolan74
Skype: egolan74

P  Save a tree. Please don't print this e-mail unless it's really necessary

Eyal,
 Did you check out the spring data project?

 http://www.springsource.org/spring-data/hadoop

 That might offer some help.

thanks
mahadev

On Mon, Dec 26, 2011 at 2:12 AM, Eyal Golan <[EMAIL PROTECTED]> wrote:

> Hi everyone,
> This is my first post in this list, as I am a newb with Hadoop.
> I am looking in the web for some documentation and example on how to use
> DI framework with Hadoop.
>
> Basically I want to have the mapping / reducing logic outside the mapper /
> reducer classes.
>
> I did a small prototype to show POC ofr our project.
> In the prototype I had classes with static methods that did the mapping /
> reducing logic.
> this is not my preferable way of course.
>
> So,
> Are there any recommendations for good DI in Hadoop documentation?
>
> Thanks,
>
> Eyal
>
>
> Eyal Golan
> [EMAIL PROTECTED]
>
> Visit: http://jvdrums.sourceforge.net/
> LinkedIn: http://www.linkedin.com/in/egolan74
> Skype: egolan74
>
> P  Save a tree. Please don't print this e-mail unless it's really necessary
>
>

Thanks.
I did see it before.
Will check it a little bit more.
Eyal Golan
[EMAIL PROTECTED]

Visit: http://jvdrums.sourceforge.net/
LinkedIn: http://www.linkedin.com/in/egolan74
Skype: egolan74

P  Save a tree. Please don't print this e-mail unless it's really necessary

On Tue, Dec 27, 2011 at 2:04 AM, Mahadev Konar <[EMAIL PROTECTED]>wrote:

> Eyal,
>  Did you check out the spring data project?
>
>  http://www.springsource.org/spring-data/hadoop
>
>  That might offer some help.
>
> thanks
> mahadev
>
>
> On Mon, Dec 26, 2011 at 2:12 AM, Eyal Golan <[EMAIL PROTECTED]> wrote:
>
>> Hi everyone,
>> This is my first post in this list, as I am a newb with Hadoop.
>> I am looking in the web for some documentation and example on how to use
>> DI framework with Hadoop.
>>
>> Basically I want to have the mapping / reducing logic outside the mapper
>> / reducer classes.
>>
>> I did a small prototype to show POC ofr our project.
>> In the prototype I had classes with static methods that did the mapping /
>> reducing logic.
>> this is not my preferable way of course.
>>
>> So,
>> Are there any recommendations for good DI in Hadoop documentation?
>>
>> Thanks,
>>
>> Eyal
>>
>>
>> Eyal Golan
>> [EMAIL PROTECTED]
>>
>> Visit: http://jvdrums.sourceforge.net/
>> LinkedIn: http://www.linkedin.com/in/egolan74
>> Skype: egolan74
>>
>> P  Save a tree. Please don't print this e-mail unless it's really
>> necessary
>>
>>
>

Hi,

My name is Costin Leau and I'm the lead of Spring Hadoop (SHDP) project.

SHDP provides DI support allowing basic POJOs to be used as
mapper/reducers. This feature is currently developed on a dedicated
branch [1] and we plan to merge it in master in the near future.

In addition to the pojo/DI support, you might be interested in the SHDP
integration with HBase/Hive/Pig - see [2]

Everything is open-source, under ASL 2, available on github. We plan to
release the milestone in Jan/Feb 2012.

Feedback and ideas are more then welcome!

[1] https://github.com/SpringSource/spring-hadoop/tree/mr-pojo
[2] http://static.springsource.org/spring-hadoop/docs/current/reference/

Hope this helps,
--
Costin

Thank you Costin.

I have worked with Spring before but never set up a project.
So I guess I'll need to do a little bit more ramp-up to use Spring-Hadoop.

Another issue that concerns me is the Hadoop cluster management.
I don't have access for the cluster, besides adding my job and use some
kind of CLI to schedule / trigger the jobs.

I am wondering whether I can create a Hadoop-Spring application without
managing the cluster.
Thanks,

Eyal

Eyal Golan
[EMAIL PROTECTED]

Visit: http://jvdrums.sourceforge.net/
LinkedIn: http://www.linkedin.com/in/egolan74
Skype: egolan74

P  Save a tree. Please don't print this e-mail unless it's really necessary

On Fri, Dec 30, 2011 at 11:53 AM, Costin Leau <[EMAIL PROTECTED]> wrote:

> Hi,
>
> My name is Costin Leau and I'm the lead of Spring Hadoop (SHDP) project.
>
> SHDP provides DI support allowing basic POJOs to be used as
> mapper/reducers. This feature is currently developed on a dedicated
> branch [1] and we plan to merge it in master in the near future.
>
> In addition to the pojo/DI support, you might be interested in the SHDP
> integration with HBase/Hive/Pig - see [2]
>
> Everything is open-source, under ASL 2, available on github. We plan to
> release the milestone in Jan/Feb 2012.
>
> Feedback and ideas are more then welcome!
>
> [1] https://github.com/SpringSource/spring-hadoop/tree/mr-pojo
> [2] http://static.springsource.org/spring-hadoop/docs/current/reference/
>
> Hope this helps,
> --
> Costin
>

Hi,

This tends to be a problem with any library that you use inside Hadoop.

One of the easiest way to distribute your dependencies (w/o managing) is
to put them under lib/ folder in your job and Hadoop will take care of
the rest - if there are any better suggestions, I'd be interested in
hearing about them.

Cheers,

On 12/30/2011 12:56 PM, Eyal Golan wrote:
> Thank you Costin.
>
> I have worked with Spring before but never set up a project.
> So I guess I'll need to do a little bit more ramp-up to use Spring-Hadoop.
>
> Another issue that concerns me is the Hadoop cluster management.
> I don't have access for the cluster, besides adding my job and use some
> kind of CLI to schedule / trigger the jobs.
>
> I am wondering whether I can create a Hadoop-Spring application without
> managing the cluster.
>
>
> Thanks,
>
> Eyal
>
> Eyal Golan
> [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
>
> Visit: http://jvdrums.sourceforge.net/
> LinkedIn: http://www.linkedin.com/in/egolan74
> Skype: egolan74
>
> P  Save a tree. Please don't print this e-mail unless it's really necessary
>
>
>
> On Fri, Dec 30, 2011 at 11:53 AM, Costin Leau <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     Hi,
>
>     My name is Costin Leau and I'm the lead of Spring Hadoop (SHDP) project.
>
>     SHDP provides DI support allowing basic POJOs to be used as
>     mapper/reducers. This feature is currently developed on a dedicated
>     branch [1] and we plan to merge it in master in the near future.
>
>     In addition to the pojo/DI support, you might be interested in the SHDP
>     integration with HBase/Hive/Pig - see [2]
>
>     Everything is open-source, under ASL 2, available on github. We plan to
>     release the milestone in Jan/Feb 2012.
>
>     Feedback and ideas are more then welcome!
>
>     [1] https://github.com/SpringSource/spring-hadoop/tree/mr-pojo
>     [2] http://static.springsource.org/spring-hadoop/docs/current/reference/
>
>     Hope this helps,
>     --
>     Costin
>
>
--
Costin

Using guice in your MapReduce task is easy. Just create the injector in the
configure/setup method and include the jars in the distributed cache.

-- Owen

On Dec 27, 2011, at 4:01 PM, Eyal Golan <[EMAIL PROTECTED]> wrote:

Thanks.
I did see it before.
Will check it a little bit more.
Eyal Golan
[EMAIL PROTECTED]

Visit: http://jvdrums.sourceforge.net/
LinkedIn: http://www.linkedin.com/in/egolan74
Skype: egolan74

P  Save a tree. Please don't print this e-mail unless it's really necessary

On Tue, Dec 27, 2011 at 2:04 AM, Mahadev Konar <[EMAIL PROTECTED]>wrote:

> Eyal,
>  Did you check out the spring data project?
>
>  http://www.springsource.org/spring-data/hadoop
>
>  That might offer some help.
>
> thanks
> mahadev
>
>
> On Mon, Dec 26, 2011 at 2:12 AM, Eyal Golan <[EMAIL PROTECTED]> wrote:
>
>> Hi everyone,
>> This is my first post in this list, as I am a newb with Hadoop.
>> I am looking in the web for some documentation and example on how to use
>> DI framework with Hadoop.
>>
>> Basically I want to have the mapping / reducing logic outside the mapper
>> / reducer classes.
>>
>> I did a small prototype to show POC ofr our project.
>> In the prototype I had classes with static methods that did the mapping /
>> reducing logic.
>> this is not my preferable way of course.
>>
>> So,
>> Are there any recommendations for good DI in Hadoop documentation?
>>
>> Thanks,
>>
>> Eyal
>>
>>
>> Eyal Golan
>> [EMAIL PROTECTED]
>>
>> Visit: http://jvdrums.sourceforge.net/
>> LinkedIn: http://www.linkedin.com/in/egolan74
>> Skype: egolan74
>>
>> P  Save a tree. Please don't print this e-mail unless it's really
>> necessary
>>
>>
>