Hello,

I have a weird issue, I am using sqoop to import data from MySQL into
HBase, sqoop confirms that 2.5 million records were imported, when I do
count "table_name" in HBase shell it returns numbers like:

260970 row(s) in 20.4740 seconds
>

(I have used sqoop to import same data from mysql to hive and it worked
smoothly without any problem)

when I run the count command again the number may change without any change
to the table, the other weird issue, if I reboot the server and run HBase
shell all the tables will be disappeared and I have to create them again.

I created a Hive external table from one of these HBase tables, when I run
a query like:

select count(1) from table_name;
it returns a number that is very different from the count of HBase command.

Any idea what is going and where the problem is?

Thanks in advanced.

--
Ibrahim

Hi Ibrahim,

Thanks for reporting the issue you are seeing. Would you be able to provide
a little more information about the version of HBase and Sqoop that you are
using?

Also, have you checked in HDFS to see If your data is there after reboot?
-Aleks S.
On Monday, January 14, 2013, Ibrahim Yakti wrote:

> Hello,
>
> I have a weird issue, I am using sqoop to import data from MySQL into
> HBase, sqoop confirms that 2.5 million records were imported, when I do
> count "table_name" in HBase shell it returns numbers like:
>
> 260970 row(s) in 20.4740 seconds
> >
>
> (I have used sqoop to import same data from mysql to hive and it worked
> smoothly without any problem)
>
> when I run the count command again the number may change without any change
> to the table, the other weird issue, if I reboot the server and run HBase
> shell all the tables will be disappeared and I have to create them again.
>
> I created a Hive external table from one of these HBase tables, when I run
> a query like:
>
> select count(1) from table_name;
>
>
> it returns a number that is very different from the count of HBase command.
>
> Any idea what is going and where the problem is?
>
> Thanks in advanced.
>
> --
> Ibrahim
>
--
Best Regards,

Aleks Shulman
847.814.5804
Cloudera

On Mon, Jan 14, 2013 at 6:12 AM, Ibrahim Yakti <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I have a weird issue, I am using sqoop to import data from MySQL into
> HBase, sqoop confirms that 2.5 million records were imported, when I do
> count "table_name" in HBase shell it returns numbers like:
>
> 260970 row(s) in 20.4740 seconds
> >
>
> (I have used sqoop to import same data from mysql to hive and it worked
> smoothly without any problem)
>
> when I run the count command again the number may change without any change
> to the table, the other weird issue, if I reboot the server and run HBase
> shell all the tables will be disappeared and I have to create them again.
>
>
Default location for hbase data is /tmp unless you change it so a reboot
will erase all you loaded into hbase.   See
http://hbase.apache.org/book.html#quickstart
Any idea what is going and where the problem is?
>

Are you overwriting old data with new?

Start small and work your way up to bigger numbers.  You might get a clue
as to where the data is going.

St.Ack

I'm using cdh4 on ec2

HBase 0.92
Sqoop 1.4.2

I'll double check versions tomorrow.

when I reboot all the tables are deleted, I'll check the default location
tomorrow as well.

What about the weird count issue?

Thanks,
Ibrahim

Sent from another galaxy device.
On Jan 14, 2013 8:22 PM, "Stack" <[EMAIL PROTECTED]> wrote:

> On Mon, Jan 14, 2013 at 6:12 AM, Ibrahim Yakti <[EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > I have a weird issue, I am using sqoop to import data from MySQL into
> > HBase, sqoop confirms that 2.5 million records were imported, when I do
> > count "table_name" in HBase shell it returns numbers like:
> >
> > 260970 row(s) in 20.4740 seconds
> > >
> >
> > (I have used sqoop to import same data from mysql to hive and it worked
> > smoothly without any problem)
> >
> > when I run the count command again the number may change without any
> change
> > to the table, the other weird issue, if I reboot the server and run HBase
> > shell all the tables will be disappeared and I have to create them again.
> >
> >
> Default location for hbase data is /tmp unless you change it so a reboot
> will erase all you loaded into hbase.   See
> http://hbase.apache.org/book.html#quickstart
>
>
> Any idea what is going and where the problem is?
> >
>
> Are you overwriting old data with new?
>
> Start small and work your way up to bigger numbers.  You might get a clue
> as to where the data is going.
>
> St.Ack
>

What is the root directory location of hbase 
Sent from Samsung Galaxy NoteIbrahim Yakti <[EMAIL PROTECTED]> wrote:I'm using cdh4 on ec2

HBase 0.92
Sqoop 1.4.2

I'll double check versions tomorrow.

when I reboot all the tables are deleted, I'll check the default location
tomorrow as well.

What about the weird count issue?

Thanks,
Ibrahim

Sent from another galaxy device.
On Jan 14, 2013 8:22 PM, "Stack" <[EMAIL PROTECTED]> wrote:

> On Mon, Jan 14, 2013 at 6:12 AM, Ibrahim Yakti <[EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > I have a weird issue, I am using sqoop to import data from MySQL into
> > HBase, sqoop confirms that 2.5 million records were imported, when I do
> > count "table_name" in HBase shell it returns numbers like:
> >
> > 260970 row(s) in 20.4740 seconds
> > >
> >
> > (I have used sqoop to import same data from mysql to hive and it worked
> > smoothly without any problem)
> >
> > when I run the count command again the number may change without any
> change
> > to the table, the other weird issue, if I reboot the server and run HBase
> > shell all the tables will be disappeared and I have to create them again.
> >
> >
> Default location for hbase data is /tmp unless you change it so a reboot
> will erase all you loaded into hbase.   See
> http://hbase.apache.org/book.html#quickstart
>
>
> Any idea what is going and where the problem is?
> >
>
> Are you overwriting old data with new?
>
> Start small and work your way up to bigger numbers.  You might get a clue
> as to where the data is going.
>
> St.Ack
>

HBase Version: 0.92.1-cdh4.1.2, rUnknown, Thu Nov  1 18:03:50 PDT 2012
Sqoop Version: 1.4.1-cdh4.1.2

Stack: yes, it seems that hbase default root directory is /tmp working to
fix this now.

I am writing the data for the first time into HBase from MySQL, not
overwriting it.

--
Ibrahim
On Mon, Jan 14, 2013 at 10:44 PM, Shashwat Shriparv <
[EMAIL PROTECTED]> wrote:

> What is the root directory location of hbase
>
>
>
>
> Sent from Samsung Galaxy NoteIbrahim Yakti <[EMAIL PROTECTED]> wrote:I'm
> using cdh4 on ec2
>
> HBase 0.92
> Sqoop 1.4.2
>
> I'll double check versions tomorrow.
>
> when I reboot all the tables are deleted, I'll check the default location
> tomorrow as well.
>
> What about the weird count issue?
>
> Thanks,
> Ibrahim
>
> Sent from another galaxy device.
> On Jan 14, 2013 8:22 PM, "Stack" <[EMAIL PROTECTED]> wrote:
>
> > On Mon, Jan 14, 2013 at 6:12 AM, Ibrahim Yakti <[EMAIL PROTECTED]> wrote:
> >
> > > Hello,
> > >
> > > I have a weird issue, I am using sqoop to import data from MySQL into
> > > HBase, sqoop confirms that 2.5 million records were imported, when I do
> > > count "table_name" in HBase shell it returns numbers like:
> > >
> > > 260970 row(s) in 20.4740 seconds
> > > >
> > >
> > > (I have used sqoop to import same data from mysql to hive and it worked
> > > smoothly without any problem)
> > >
> > > when I run the count command again the number may change without any
> > change
> > > to the table, the other weird issue, if I reboot the server and run
> HBase
> > > shell all the tables will be disappeared and I have to create them
> again.
> > >
> > >
> > Default location for hbase data is /tmp unless you change it so a reboot
> > will erase all you loaded into hbase.   See
> > http://hbase.apache.org/book.html#quickstart
> >
> >
> > Any idea what is going and where the problem is?
> > >
> >
> > Are you overwriting old data with new?
> >
> > Start small and work your way up to bigger numbers.  You might get a clue
> > as to where the data is going.
> >
> > St.Ack
> >
>

The root directory is set to /var/lib/hbase/

The count issue is still exist:

*Sqoop: *
13/01/15 08:55:23 INFO mapreduce.ImportJobBase: Retrieved 1754285 records.

*MySQL:*
+----------+
| count(1) |
+----------+
|  1754303 |
+----------+
*HBase:*
104210 row(s) in 11.4980 seconds

running count command five minutes later:
327230 row(s) in 18.4020 seconds

*Hive *(using HBase as external table)*:*
OK
104222
Time taken: 41.971 seconds

running same command five minutes later:
OK
454038
Time taken: 149.328 seconds
Any idea what is going on or what is missing?

--
Ibrahim
On Tue, Jan 15, 2013 at 11:44 AM, Ibrahim Yakti <[EMAIL PROTECTED]> wrote:

> HBase Version: 0.92.1-cdh4.1.2, rUnknown, Thu Nov  1 18:03:50 PDT 2012
> Sqoop Version: 1.4.1-cdh4.1.2
>
> Stack: yes, it seems that hbase default root directory is /tmp working to
> fix this now.
>
> I am writing the data for the first time into HBase from MySQL, not
> overwriting it.
>
>
>
> --
> Ibrahim
>
>
> On Mon, Jan 14, 2013 at 10:44 PM, Shashwat Shriparv <
> [EMAIL PROTECTED]> wrote:
>
>> What is the root directory location of hbase
>>
>>
>>
>>
>> Sent from Samsung Galaxy NoteIbrahim Yakti <[EMAIL PROTECTED]> wrote:I'm
>> using cdh4 on ec2
>>
>> HBase 0.92
>> Sqoop 1.4.2
>>
>> I'll double check versions tomorrow.
>>
>> when I reboot all the tables are deleted, I'll check the default location
>> tomorrow as well.
>>
>> What about the weird count issue?
>>
>> Thanks,
>> Ibrahim
>>
>> Sent from another galaxy device.
>> On Jan 14, 2013 8:22 PM, "Stack" <[EMAIL PROTECTED]> wrote:
>>
>> > On Mon, Jan 14, 2013 at 6:12 AM, Ibrahim Yakti <[EMAIL PROTECTED]> wrote:
>> >
>> > > Hello,
>> > >
>> > > I have a weird issue, I am using sqoop to import data from MySQL into
>> > > HBase, sqoop confirms that 2.5 million records were imported, when I
>> do
>> > > count "table_name" in HBase shell it returns numbers like:
>> > >
>> > > 260970 row(s) in 20.4740 seconds
>> > > >
>> > >
>> > > (I have used sqoop to import same data from mysql to hive and it
>> worked
>> > > smoothly without any problem)
>> > >
>> > > when I run the count command again the number may change without any
>> > change
>> > > to the table, the other weird issue, if I reboot the server and run
>> HBase
>> > > shell all the tables will be disappeared and I have to create them
>> again.
>> > >
>> > >
>> > Default location for hbase data is /tmp unless you change it so a reboot
>> > will erase all you loaded into hbase.   See
>> > http://hbase.apache.org/book.html#quickstart
>> >
>> >
>> > Any idea what is going and where the problem is?
>> > >
>> >
>> > Are you overwriting old data with new?
>> >
>> > Start small and work your way up to bigger numbers.  You might get a
>> clue
>> > as to where the data is going.
>> >
>> > St.Ack
>> >
>>
>
>

On Tue, Jan 15, 2013 at 1:07 AM, Ibrahim Yakti <[EMAIL PROTECTED]> wrote:

> The root directory is set to /var/lib/hbase/
>
> The count issue is still exist:
>
> *Sqoop: *
> 13/01/15 08:55:23 INFO mapreduce.ImportJobBase: Retrieved 1754285 records.
>
> *MySQL:*
> +----------+
> | count(1) |
> +----------+
> |  1754303 |
> +----------+
>
>
> *HBase:*
> 104210 row(s) in 11.4980 seconds
>
> running count command five minutes later:
> 327230 row(s) in 18.4020 seconds
>
>
>
> *Hive *(using HBase as external table)*:*
> OK
> 104222
> Time taken: 41.971 seconds
>
> running same command five minutes later:
> OK
> 454038
> Time taken: 149.328 seconds
>
>
> Any idea what is going on or what is missing?
>
>

Upload is still happening?

HBase does not usually lose data so I'd say the issue has to do w/ how you
are doing your upload or it is some characteristic of your data -- or even
of the schema you have chosen for hbase.

Try with a little data first... a subselect.  Confirm that works and go up
from there.  If data is missing, try figure what is missing and if the
missing data has a common characteristic.

Good luck,
St.Ack

The upload finished and sqoop confirmed it was done successfully. I use
sqoop to create the table and schema.

I've tried the same command and same data on Amazon EMR instance and it
worked without any problem.

If it's in schema why the count changes when you run the count command a
few minutes later?

What do you think?
On Jan 15, 2013 8:05 PM, "Stack" <[EMAIL PROTECTED]> wrote:

> On Tue, Jan 15, 2013 at 1:07 AM, Ibrahim Yakti <[EMAIL PROTECTED]> wrote:
>
> > The root directory is set to /var/lib/hbase/
> >
> > The count issue is still exist:
> >
> > *Sqoop: *
> > 13/01/15 08:55:23 INFO mapreduce.ImportJobBase: Retrieved 1754285
> records.
> >
> > *MySQL:*
> > +----------+
> > | count(1) |
> > +----------+
> > |  1754303 |
> > +----------+
> >
> >
> > *HBase:*
> > 104210 row(s) in 11.4980 seconds
> >
> > running count command five minutes later:
> > 327230 row(s) in 18.4020 seconds
> >
> >
> >
> > *Hive *(using HBase as external table)*:*
> > OK
> > 104222
> > Time taken: 41.971 seconds
> >
> > running same command five minutes later:
> > OK
> > 454038
> > Time taken: 149.328 seconds
> >
> >
> > Any idea what is going on or what is missing?
> >
> >
>
> Upload is still happening?
>
> HBase does not usually lose data so I'd say the issue has to do w/ how you
> are doing your upload or it is some characteristic of your data -- or even
> of the schema you have chosen for hbase.
>
> Try with a little data first... a subselect.  Confirm that works and go up
> from there.  If data is missing, try figure what is missing and if the
> missing data has a common characteristic.
>
> Good luck,
> St.Ack
>