Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Does Sqoop support importing non-relational data to Hadoop


Copy link to this message
-
Re: Does Sqoop support importing non-relational data to Hadoop
We have used flume to collect files from windows using old flume (0.94, I
believe).  There is a flume windows agent.  It was working well but did not
scale well for us.  When I tested collecting from windows 5 servers, it
worked well.  With like 20, collector would fail ans stop collecting data.

There is no windows agent in flume-ng (may be there now) when I checked few
months ago.

On Wed, Nov 28, 2012 at 10:45 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Jia,
> thank you very much for your questions. Sqoop is designed as a batch tool
> to
> transfer data from database and warehouse systems to Hadoop ecosystem and
> vice-versa. Right now, Sqoop is supporting only JDBC compliant databases.
> This
> requirement will however fade away with Sqoop 2.
>
> Based on your questions, it seems to me that you're more looking for an
> online
> ingest system rather than batch one. In such case, I would recommend
> checking
> Apache Flume project [1] that aim to address online data ingesting issue.
>
> Links:
> 1: http://flume.apache.org/
>
> On Wed, Nov 28, 2012 at 02:19:57PM +0800, jia jimin wrote:
> > Hi there,
> >
> > I am investigating Sqoop on windows for importing data to HDFS and have
> > some questions :
> >
> > 1. Does Sqoop support importing non-relational data such as event log or
> > text file to HDFS  ?
> >
> > 2. If our client machine changed frequently ( recycle old machines and
> add
> > new machines) ,   Can sqoop automatically import data by changing some
> > configurations dynamically ?
> >
> > Thanks for looking at these questions !
> >
> > Regards
> > Benjamin
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB