Dear MapReduce User Groups:
We want to process large amount of videos (typically 30 days old storage with size around 1TB) using Hadoop.
Can somebody point me to code samples or classes which can take video files in its original compressed format (H.264, MPEG-4) and then process using Mappers?
Thanks and Regards,

~~
Dr. Rajen Bhatt
(Corporate Research @ Robert Bosch, India)
Off: +91-80-4191-6699
Mob: +91-9901241005

This link might help you...
example<http://musicmachinery.com/2011/09/04/how-to-process-a-million-songs-in-20-minutes/>

On Tue, Sep 20, 2011 at 9:52 AM, Rajen Bhatt (RBEI/EST1) <
[EMAIL PROTECTED]> wrote:

>  Dear MapReduce User Groups:
> We want to process large amount of videos (typically 30 days old storage
> with size around 1TB) using Hadoop.
> Can somebody point me to code samples or classes which can take video files
> in its original compressed format (H.264, MPEG-4) and then process using
> Mappers?
> Thanks and Regards,
>
> ~~
> *Dr. Rajen Bhatt *
> *(**Corporate Research** **@ Robert Bosch, India**)*
> Off: +91-80-4191-6699
> Mob: +91-9901241005
>
>
>
>

--
Regards,
Swathi.V.

Another thing to think about is that you may not need to split the videos at all.  If you have lots of video files instead of a few big ones, and each can more or less be processed independently then you can use something like nline input format, not nline itself necessarily but something like it, to process each video separately.  You would have to write the code to read in the video file, but there are APIs to do that, like OpenCV.  This is what I did in the past to train and score machine learned classifiers on image and video files using Hadoop.

--Bobby Evans

On 9/19/11 11:54 PM, "Swathi V" <[EMAIL PROTECTED]> wrote:

This link might help you...
example <http://musicmachinery.com/2011/09/04/how-to-process-a-million-songs-in-20-minutes/>

On Tue, Sep 20, 2011 at 9:52 AM, Rajen Bhatt (RBEI/EST1) <[EMAIL PROTECTED]> wrote:
Dear MapReduce User Groups:
We want to process large amount of videos (typically 30 days old storage with size around 1TB) using Hadoop.
Can somebody point me to code samples or classes which can take video files in its original compressed format (H.264, MPEG-4) and then process using Mappers?
Thanks and Regards,

~~
Dr. Rajen Bhatt
(Corporate Research @ Robert Bosch, India)
Off: +91-80-4191-6699
Mob: +91-9901241005
[cid:3399368505_4188812]

Dear Bobby:
Thanks for your reply.
My problem is that I have just one file, which is very large typically in terabytes or petabytes and then I want to split them, feed to mappers, and process on worker nodes. Is some example class is available with somebody, it shall help the student.
Thanks and Regards,

~~
Dr. Rajen Bhatt
(Corporate Research @ Robert Bosch, India)
Off: +91-80-4191-6699
Mob: +91-9901241005
[cid:[EMAIL PROTECTED]4B70]

From: Robert Evans [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, 20. September 2011 11:32 PM
To: [EMAIL PROTECTED]
Subject: Re: Using HADOOP for Processing Videos

Another thing to think about is that you may not need to split the videos at all.  If you have lots of video files instead of a few big ones, and each can more or less be processed independently then you can use something like nline input format, not nline itself necessarily but something like it, to process each video separately.  You would have to write the code to read in the video file, but there are APIs to do that, like OpenCV.  This is what I did in the past to train and score machine learned classifiers on image and video files using Hadoop.

--Bobby Evans

On 9/19/11 11:54 PM, "Swathi V" <[EMAIL PROTECTED]> wrote:
This link might help you...
example <http://musicmachinery.com/2011/09/04/how-to-process-a-million-songs-in-20-minutes/>

On Tue, Sep 20, 2011 at 9:52 AM, Rajen Bhatt (RBEI/EST1) <[EMAIL PROTECTED]> wrote:
Dear MapReduce User Groups:
We want to process large amount of videos (typically 30 days old storage with size around 1TB) using Hadoop.
Can somebody point me to code samples or classes which can take video files in its original compressed format (H.264, MPEG-4) and then process using Mappers?
Thanks and Regards,

~~
Dr. Rajen Bhatt
(Corporate Research @ Robert Bosch, India)
Off: +91-80-4191-6699
Mob: +91-9901241005
[cid:[EMAIL PROTECTED]4B70]

Dr. Rajen Bhatt,

I don't now of any example code that will split a video file for Hadoop.  If you write one then please start a git-hub project or something like it and let us know because I can see it being extremely useful for others wanting to do multimedia processing.

You might want to look at ffmpeg(libavformat/libavcodec)  I am not sure how this would work completely, but in the splitter you could open up the file and get out the file's length in total time or frames and divided it up per mapper, then in each mapper have libavformat/libavcodec open up the file again and seek to their start time or frame.  I am not sure how efficient libavformat/libavcodec are in seeking.  It would be bad if they actually read the entire file to get to a given point.  It would also be good if the in the splitter you could try and get what the offset into the file is for a given time or frame so that you can give hits to Hadoop to know where to place the mapper.  This also assumes that you can set up something to bridge HDFS to libavformat probably using JNI.

--Bobby Evans
On 9/20/11 11:12 PM, "Rajen Bhatt (RBEI/EST1)" <[EMAIL PROTECTED]> wrote:

Dear Bobby:
Thanks for your reply.
My problem is that I have just one file, which is very large typically in terabytes or petabytes and then I want to split them, feed to mappers, and process on worker nodes. Is some example class is available with somebody, it shall help the student.
Thanks and Regards,
~~
Dr. Rajen Bhatt
(Corporate Research @ Robert Bosch, India)
Off: +91-80-4191-6699
Mob: +91-9901241005
[cid:3399453662_6132054]
From: Robert Evans [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, 20. September 2011 11:32 PM
To: [EMAIL PROTECTED]
Subject: Re: Using HADOOP for Processing Videos

Another thing to think about is that you may not need to split the videos at all.  If you have lots of video files instead of a few big ones, and each can more or less be processed independently then you can use something like nline input format, not nline itself necessarily but something like it, to process each video separately.  You would have to write the code to read in the video file, but there are APIs to do that, like OpenCV.  This is what I did in the past to train and score machine learned classifiers on image and video files using Hadoop.

--Bobby Evans

On 9/19/11 11:54 PM, "Swathi V" <[EMAIL PROTECTED]> wrote:
This link might help you...
example <http://musicmachinery.com/2011/09/04/how-to-process-a-million-songs-in-20-minutes/>

On Tue, Sep 20, 2011 at 9:52 AM, Rajen Bhatt (RBEI/EST1) <[EMAIL PROTECTED]> wrote:
Dear MapReduce User Groups:
We want to process large amount of videos (typically 30 days old storage with size around 1TB) using Hadoop.
Can somebody point me to code samples or classes which can take video files in its original compressed format (H.264, MPEG-4) and then process using Mappers?
Thanks and Regards,

~~
Dr. Rajen Bhatt
(Corporate Research @ Robert Bosch, India)
Off: +91-80-4191-6699
Mob: +91-9901241005
[cid:3399453662_6130082]

Hello Rajen,

Ideally you just need to write your appropriate InputFormat derivative.
Here's my colleague Eric Sammer talking about the same things I would, so
I'll just link to his response:
http://stackoverflow.com/questions/3044050/image-processing-with-hadoop

I'm not aware of an open source library that tends specifically to
hadoop+video (others may pitch in if they know) but you can reutilize CLI
tools (such as mplayer) via hadoop-streaming and cut down on writing code in
many cases.

On Tue, Sep 20, 2011 at 9:52 AM, Rajen Bhatt (RBEI/EST1) <
[EMAIL PROTECTED]> wrote:

>  Dear MapReduce User Groups:
> We want to process large amount of videos (typically 30 days old storage
> with size around 1TB) using Hadoop.
> Can somebody point me to code samples or classes which can take video files
> in its original compressed format (H.264, MPEG-4) and then process using
> Mappers?
> Thanks and Regards,
>
> ~~
> *Dr. Rajen Bhatt *
> *(**Corporate Research** **@ Robert Bosch, India**)*
> Off: +91-80-4191-6699
> Mob: +91-9901241005
>
>
>
>

--
Harsh J

Hi Rajen,

 

            You can write a mapred application, which uses the customized
InputFormat and OutputFormat(If you want to produce output as video files).
This customized InputFormat can be written in such a way that it can
understand your video files and split accordingly.

 

 

You can go through this link for more info on input format and out format.

 

 
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/In
putFormat.html

 

 
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Ou
tputFormat.html

 

Devaraj K

 

  _____  

From: Rajen Bhatt (RBEI/EST1) [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, September 20, 2011 9:53 AM
To: [EMAIL PROTECTED]
Subject: Using HADOOP for Processing Videos

 

Dear MapReduce User Groups:

We want to process large amount of videos (typically 30 days old storage
with size around 1TB) using Hadoop.

Can somebody point me to code samples or classes which can take video files
in its original compressed format (H.264, MPEG-4) and then process using
Mappers?

Thanks and Regards,

 

~~

Dr. Rajen Bhatt

(Corporate Research @ Robert Bosch, India)

Off: +91-80-4191-6699

Mob: +91-9901241005