Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # user - Does Sqoop support importing non-relational data to Hadoop


Copy link to this message
-
Re: Does Sqoop support importing non-relational data to Hadoop
Chalcy 2012-11-28, 16:03
We have used flume to collect files from windows using old flume (0.94, I
believe).  There is a flume windows agent.  It was working well but did not
scale well for us.  When I tested collecting from windows 5 servers, it
worked well.  With like 20, collector would fail ans stop collecting data.

There is no windows agent in flume-ng (may be there now) when I checked few
months ago.

On Wed, Nov 28, 2012 at 10:45 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Jia,
> thank you very much for your questions. Sqoop is designed as a batch tool
> to
> transfer data from database and warehouse systems to Hadoop ecosystem and
> vice-versa. Right now, Sqoop is supporting only JDBC compliant databases.
> This
> requirement will however fade away with Sqoop 2.
>
> Based on your questions, it seems to me that you're more looking for an
> online
> ingest system rather than batch one. In such case, I would recommend
> checking
> Apache Flume project [1] that aim to address online data ingesting issue.
>
> Links:
> 1: http://flume.apache.org/
>
> On Wed, Nov 28, 2012 at 02:19:57PM +0800, jia jimin wrote:
> > Hi there,
> >
> > I am investigating Sqoop on windows for importing data to HDFS and have
> > some questions :
> >
> > 1. Does Sqoop support importing non-relational data such as event log or
> > text file to HDFS  ?
> >
> > 2. If our client machine changed frequently ( recycle old machines and
> add
> > new machines) ,   Can sqoop automatically import data by changing some
> > configurations dynamically ?
> >
> > Thanks for looking at these questions !
> >
> > Regards
> > Benjamin
>