More info on this issue:

1- I use PIG 0.8.1 and HBase 0.90.3 and Hadoop 0.20-append
2- The issue can be reproduced with PIG trunk too

The script:

start_sessions = LOAD 'startSession.mde253811.preprod.ubithere.com'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
meta:infoid meta:imei meta:timestamp') AS (sid:chararray,
infoid:chararray, imei:chararray, start:long);
end_sessions = LOAD 'endSession.mde253811.preprod.ubithere.com'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
meta:timestamp meta:locid') AS (sid:chararray, end:long,
locid:chararray);
sessions = JOIN start_sessions BY sid, end_sessions BY sid;
sessions = FILTER sessions BY end > start AND end - start < 86400000L;
sessions = FOREACH sessions GENERATE start_sessions::sid, imei,
start, end;
sessions = LIMIT sessions 100;
dump sessions;
<output 1>
dump sessions;
<output 2>

The issue:

<output 1> is empty
<output 2> is 100 lines

I can reproduce the issue systematically.

Please advice: this issue prevent me from moving to HBase 0.90.3 in
production, as I need to upgrade to PIG 0.8.1 at the same time !

A precision: HBase classes of the PIG trunk cannot be compiled
inside PIG 0.8.1, so I was enable to test if a fix was introduced in
the last version of these classes.
So 2- must not be taken into account

Le 27/07/11 14:38, Vincent Barat a �crit :
> More info on this issue:
>
> 1- I use PIG 0.8.1 and HBase 0.90.3 and Hadoop 0.20-append
> 2- The issue can be reproduced with PIG trunk too
>
> The script:
>
> start_sessions = LOAD
> 'startSession.mde253811.preprod.ubithere.com' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
> meta:infoid meta:imei meta:timestamp') AS (sid:chararray,
> infoid:chararray, imei:chararray, start:long);
> end_sessions = LOAD 'endSession.mde253811.preprod.ubithere.com'
> USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
> meta:timestamp meta:locid') AS (sid:chararray, end:long,
> locid:chararray);
> sessions = JOIN start_sessions BY sid, end_sessions BY sid;
> sessions = FILTER sessions BY end > start AND end - start <
> 86400000L;
> sessions = FOREACH sessions GENERATE start_sessions::sid, imei,
> start, end;
> sessions = LIMIT sessions 100;
> dump sessions;
> <output 1>
> dump sessions;
> <output 2>
>
> The issue:
>
> <output 1> is empty
> <output 2> is 100 lines
>
> I can reproduce the issue systematically.
>
> Please advice: this issue prevent me from moving to HBase 0.90.3
> in production, as I need to upgrade to PIG 0.8.1 at the same time !
>
>

--

*Vincent BARAT, UBIKOD, CTO*
[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>  Mob +33 (0)6 15 41 15 18

UBIKOD Paris, c/o ESSEC VENTURES, Avenue Bernard Hirsch, 95021
Cergy-Pontoise cedex, FRANCE, Tel +33 (0)1 34 43 28 89

UBIKOD Rennes, 10 rue Duhamel, 35000 Rennes, FRANCE, Tel. +33 (0)2
99 65 69 13
www.ubikod.com <http://www.ubikod.com/>@ubikod
<http://twitter.com/ubikod>

www.capptain.com <http://www.capptain.com/>@capptain_hq
<http://twitter.com/capptain_hq>
IMPORTANT NOTICE -- UBIKOD and CAPPTAIN are registered trademarks of
UBIKOD S.A.R.L., all copyrights are reserved.  The contents of this
email and attachments are confidential and may be subject to legal
privilege and/or protected by copyright. Copying or communicating
any part of it to others is prohibited and may be unlawful. If you
are not the intended recipient you must not use, copy, distribute or
rely on this email and should please return it immediately or notify
us by telephone. At present the integrity of email across the
Internet cannot be guaranteed. Therefore UBIKOD S.A.R.L. will not
accept liability for any claims arising as a result of the use of
this medium for transmissions by or to UBIKOD S.A.R.L.. UBIKOD
S.A.R.L. may exercise any of its rights under relevant law, to
monitor the content of all electronic communications. You should
therefore be aware that this communication and any responses might
have been monitored, and may be accessed by UBIKOD S.A.R.L. The
views expressed in this document are that of the individual and may
not necessarily constitute or imply its endorsement or
recommendation by UBIKOD S.A.R.L. The content of this electronic
mail may be subject to the confidentiality terms of a
"Non-Disclosure Agreement" (NDA).

I built the pig trunk with hbase 0.90.3 client lib (ant
-Dhbase.version=0.90.3) and the issue is still here.

It makes me thing about an issue in the optimizer... Anyway the fact
is that my request is not complex, so I wonder how such an issue can
go through PIG test suite !

Any help ?

Le 27/07/11 14:38, Vincent Barat a �crit :
> More info on this issue:
>
> 1- I use PIG 0.8.1 and HBase 0.90.3 and Hadoop 0.20-append
> 2- The issue can be reproduced with PIG trunk too
>
> The script:
>
> start_sessions = LOAD
> 'startSession.mde253811.preprod.ubithere.com' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
> meta:infoid meta:imei meta:timestamp') AS (sid:chararray,
> infoid:chararray, imei:chararray, start:long);
> end_sessions = LOAD 'endSession.mde253811.preprod.ubithere.com'
> USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
> meta:timestamp meta:locid') AS (sid:chararray, end:long,
> locid:chararray);
> sessions = JOIN start_sessions BY sid, end_sessions BY sid;
> sessions = FILTER sessions BY end > start AND end - start <
> 86400000L;
> sessions = FOREACH sessions GENERATE start_sessions::sid, imei,
> start, end;
> sessions = LIMIT sessions 100;
> dump sessions;
> <output 1>
> dump sessions;
> <output 2>
>
> The issue:
>
> <output 1> is empty
> <output 2> is 100 lines
>
> I can reproduce the issue systematically.
>
> Please advice: this issue prevent me from moving to HBase 0.90.3
> in production, as I need to upgrade to PIG 0.8.1 at the same time !
>
>

Vincent,

is the behavior random or the same each time?

Couple of things to narrow it down..
  - attach the entire console output from PIG run when this happened.
  - only load start_sessions and end_sessions and store them..
  - load the data from tables from previous step and run the same pig
command

Consider filing a JIRA. it might be a better place to go into more details.

-Raghu.

On Wed, Jul 27, 2011 at 5:38 AM, Vincent Barat <[EMAIL PROTECTED]>wrote:

> More info on this issue:
>
> 1- I use PIG 0.8.1 and HBase 0.90.3 and Hadoop 0.20-append
> 2- The issue can be reproduced with PIG trunk too
>
> The script:
>
> start_sessions = LOAD 'startSession.mde253811.**preprod.ubithere.com<http://startSession.mde253811.preprod.ubithere.com>'
> USING org.apache.pig.backend.hadoop.**hbase.HBaseStorage('meta:sid
> meta:infoid meta:imei meta:timestamp') AS (sid:chararray, infoid:chararray,
> imei:chararray, start:long);
> end_sessions = LOAD 'endSession.mde253811.preprod.**ubithere.com<http://endSession.mde253811.preprod.ubithere.com>'
> USING org.apache.pig.backend.hadoop.**hbase.HBaseStorage('meta:sid
> meta:timestamp meta:locid') AS (sid:chararray, end:long, locid:chararray);
> sessions = JOIN start_sessions BY sid, end_sessions BY sid;
> sessions = FILTER sessions BY end > start AND end - start < 86400000L;
> sessions = FOREACH sessions GENERATE start_sessions::sid, imei, start, end;
> sessions = LIMIT sessions 100;
> dump sessions;
> <output 1>
> dump sessions;
> <output 2>
>
> The issue:
>
> <output 1> is empty
> <output 2> is 100 lines
>
> I can reproduce the issue systematically.
>
> Please advice: this issue prevent me from moving to HBase 0.90.3 in
> production, as I need to upgrade to PIG 0.8.1 at the same time !
>
>

I looked at the query plan for the query using explain, and it looks
correct.
As you said, this is a simple use case, I would be very surprised if
there is a optimizer bug here.
I suspect that something is wrong in loading the data from hbase. Are
you able to get a simple load-store script working consistently ?

Thanks,
Thejas
On 7/27/11 7:31 AM, Vincent Barat wrote:
> I built the pig trunk with hbase 0.90.3 client lib (ant
> -Dhbase.version=0.90.3) and the issue is still here.
>
> It makes me thing about an issue in the optimizer... Anyway the fact is
> that my request is not complex, so I wonder how such an issue can go
> through PIG test suite !
>
> Any help ?
>
> Le 27/07/11 14:38, Vincent Barat a �crit :
>> More info on this issue:
>>
>> 1- I use PIG 0.8.1 and HBase 0.90.3 and Hadoop 0.20-append
>> 2- The issue can be reproduced with PIG trunk too
>>
>> The script:
>>
>> start_sessions = LOAD 'startSession.mde253811.preprod.ubithere.com'
>> USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
>> meta:infoid meta:imei meta:timestamp') AS (sid:chararray,
>> infoid:chararray, imei:chararray, start:long);
>> end_sessions = LOAD 'endSession.mde253811.preprod.ubithere.com' USING
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
>> meta:timestamp meta:locid') AS (sid:chararray, end:long,
>> locid:chararray);
>> sessions = JOIN start_sessions BY sid, end_sessions BY sid;
>> sessions = FILTER sessions BY end > start AND end - start < 86400000L;
>> sessions = FOREACH sessions GENERATE start_sessions::sid, imei, start,
>> end;
>> sessions = LIMIT sessions 100;
>> dump sessions;
>> <output 1>
>> dump sessions;
>> <output 2>
>>
>> The issue:
>>
>> <output 1> is empty
>> <output 2> is 100 lines
>>
>> I can reproduce the issue systematically.
>>
>> Please advice: this issue prevent me from moving to HBase 0.90.3 in
>> production, as I need to upgrade to PIG 0.8.1 at the same time !
>>
>>

Yes: if I remove the FILTER or the JOIN clause, the loading of data
works fine and consistently.
I will do more testings, but yes, I suspect HBase loader to work
incorrectly in my case...

The same query works perfectly with HBase 0.20.6 and PIG 0.6.1.

Le 27/07/11 19:43, Thejas Nair a �crit :
> I looked at the query plan for the query using explain, and it
> looks correct.
> As you said, this is a simple use case, I would be very surprised
> if there is a optimizer bug here.
> I suspect that something is wrong in loading the data from hbase.
> Are you able to get a simple load-store script working consistently ?
>
> Thanks,
> Thejas
>
>
> On 7/27/11 7:31 AM, Vincent Barat wrote:
>> I built the pig trunk with hbase 0.90.3 client lib (ant
>> -Dhbase.version=0.90.3) and the issue is still here.
>>
>> It makes me thing about an issue in the optimizer... Anyway the
>> fact is
>> that my request is not complex, so I wonder how such an issue can go
>> through PIG test suite !
>>
>> Any help ?
>>
>> Le 27/07/11 14:38, Vincent Barat a �crit :
>>> More info on this issue:
>>>
>>> 1- I use PIG 0.8.1 and HBase 0.90.3 and Hadoop 0.20-append
>>> 2- The issue can be reproduced with PIG trunk too
>>>
>>> The script:
>>>
>>> start_sessions = LOAD 'startSession.mde253811.preprod.ubithere.com'
>>> USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
>>> meta:infoid meta:imei meta:timestamp') AS (sid:chararray,
>>> infoid:chararray, imei:chararray, start:long);
>>> end_sessions = LOAD 'endSession.mde253811.preprod.ubithere.com'
>>> USING
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
>>> meta:timestamp meta:locid') AS (sid:chararray, end:long,
>>> locid:chararray);
>>> sessions = JOIN start_sessions BY sid, end_sessions BY sid;
>>> sessions = FILTER sessions BY end > start AND end - start <
>>> 86400000L;
>>> sessions = FOREACH sessions GENERATE start_sessions::sid, imei,
>>> start,
>>> end;
>>> sessions = LIMIT sessions 100;
>>> dump sessions;
>>> <output 1>
>>> dump sessions;
>>> <output 2>
>>>
>>> The issue:
>>>
>>> <output 1> is empty
>>> <output 2> is 100 lines
>>>
>>> I can reproduce the issue systematically.
>>>
>>> Please advice: this issue prevent me from moving to HBase 0.90.3 in
>>> production, as I need to upgrade to PIG 0.8.1 at the same time !
>>>
>>>
>
>

The behavior is not random.
The first dump is always empty, and the second always works.
I will try what you ask, and if I have more details, I will create a
JIRA issue.

Thanks.

Le 27/07/11 16:59, Raghu Angadi a �crit :
> Vincent,
>
> is the behavior random or the same each time?
>
> Couple of things to narrow it down..
>    - attach the entire console output from PIG run when this happened.
>    - only load start_sessions and end_sessions and store them..
>    - load the data from tables from previous step and run the same pig
> command
>
> Consider filing a JIRA. it might be a better place to go into more details.
>
> -Raghu.
>
> On Wed, Jul 27, 2011 at 5:38 AM, Vincent Barat<[EMAIL PROTECTED]>wrote:
>
>> More info on this issue:
>>
>> 1- I use PIG 0.8.1 and HBase 0.90.3 and Hadoop 0.20-append
>> 2- The issue can be reproduced with PIG trunk too
>>
>> The script:
>>
>> start_sessions = LOAD 'startSession.mde253811.**preprod.ubithere.com<http://startSession.mde253811.preprod.ubithere.com>'
>> USING org.apache.pig.backend.hadoop.**hbase.HBaseStorage('meta:sid
>> meta:infoid meta:imei meta:timestamp') AS (sid:chararray, infoid:chararray,
>> imei:chararray, start:long);
>> end_sessions = LOAD 'endSession.mde253811.preprod.**ubithere.com<http://endSession.mde253811.preprod.ubithere.com>'
>> USING org.apache.pig.backend.hadoop.**hbase.HBaseStorage('meta:sid
>> meta:timestamp meta:locid') AS (sid:chararray, end:long, locid:chararray);
>> sessions = JOIN start_sessions BY sid, end_sessions BY sid;
>> sessions = FILTER sessions BY end>  start AND end - start<  86400000L;
>> sessions = FOREACH sessions GENERATE start_sessions::sid, imei, start, end;
>> sessions = LIMIT sessions 100;
>> dump sessions;
>> <output 1>
>> dump sessions;
>> <output 2>
>>
>> The issue:
>>
>> <output 1>  is empty
>> <output 2>  is 100 lines
>>
>> I can reproduce the issue systematically.
>>
>> Please advice: this issue prevent me from moving to HBase 0.90.3 in
>> production, as I need to upgrade to PIG 0.8.1 at the same time !
>>
>>

So, I've tried the exact same request but loading the data from HDFS
files (using the regular Pig loader) : it works !

Here is the request loading from HDFS:

start_sessions = LOAD 'start_sessions' AS (sid:chararray,
infoid:chararray, imei:chararray, start:long);
end_sessions = LOAD 'end_sessions' AS (sid:chararray, end:long,
locid:chararray);
infos = LOAD 'infos' AS (infoid:chararray, network_type:chararray,
network_subtype:chararray, locale:chararray, version_name:chararray,
carrier_country:chararray, carrier_name:chararray,
phone_manufacturer:chararray, phone_model:chararray,
firmware_version:chararray, firmware_name:chararray);
sessions = JOIN start_sessions BY sid, end_sessions BY sid;
sessions = FILTER sessions BY end > start AND end - start < 86400000L;
sessions = JOIN sessions BY infoid, infos BY infoid;
sessions = LIMIT sessions 100;
dump sessions;

The same request loading from HBase don't work:

start_sessions = LOAD 'startSession' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
meta:infoid meta:imei meta:timestamp') AS (sid:chararray,
infoid:chararray, imei:chararray, start:long);
end_sessions = LOAD 'endSession' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
meta:timestamp meta:locid') AS (sid:chararray, end:long,
locid:chararray);
infos = LOAD 'info' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:infoid
data:networkType data:networkSubtype data:locale
data:applicationVersionName data:carrierCountry data:carrierName
data:phoneManufacturer data:phoneModel data:firmwareVersion
data:firmwareName') AS (infoid:chararray, network_type:chararray,
network_subtype:chararray, locale:chararray, version_name:chararray,
carrier_country:chararray, carrier_name:chararray,
phone_manufacturer:chararray, phone_model:chararray,
firmware_version:chararray, firmware_name:chararray);
sessions = JOIN start_sessions BY sid, end_sessions BY sid;
sessions = FILTER sessions BY end > start AND end - start < 86400000L;
sessions = JOIN sessions BY infoid, infos BY infoid;
sessions = LIMIT sessions 100;
dump sessions;

I guess it definitively means there is a nasty bug in the HBase loader.

Here is the PIG dump for the non working request:

aws09:~# pig
2011-07-28 08:17:36,329 [main] INFO  org.apache.pig.Main - Logging
error messages to: /root/pig_1311841056328.log
2011-07-28 08:17:36,641 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at:
hdfs://aws09.preprod.ubithere.com:9000
2011-07-28 08:17:36,923 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to map-reduce job tracker at: aws09.preprod.ubithere.com:9001
grunt> start_sessions = LOAD
'startSession.mde253811.preprod.ubithere.com' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
meta:infoid meta:imei meta:timestamp') AS (sid:chararray,
infoid:chararray, imei:chararray, start:long);
grunt> end_sessions = LOAD
'endSession.mde253811.preprod.ubithere.com' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
meta:timestamp meta:locid') AS (sid:chararray, end:long,
locid:chararray);
grunt> infos = LOAD 'info.mde253811.preprod.ubithere.com' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:infoid
data:networkType data:networkSubtype data:locale
data:applicationVersionName data:carrierCountry data:carrierName
data:phoneManufacturer data:phoneModel data:firmwareVersion
data:firmwareName') AS (infoid:chararray, network_type:chararray,
network_subtype:chararray, locale:chararray, version_name:chararray,
carrier_country:chararray, carrier_name:chararray,
phone_manufacturer:chararray, phone_model:chararray,
firmware_version:chararray, firmware_name:chararray);
grunt> sessions = JOIN start_sessions BY sid, end_sessions BY sid;
grunt> sessions = FILTER sessions BY end > start AND end - start <
86400000L;
grunt> sessions = JOIN sessions BY infoid, infos BY infoid;
grunt> sessions = LIMIT sessions 100;
grunt> dump sessions;
2011-07-28 08:17:50,275 [main] INFO  
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: HASH_JOIN,FILTER,LIMIT
2011-07-28 08:17:50,275 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
pig.usenewlogicalplan is set to true. New logical plan will be used.
2011-07-28 08:17:51,213 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
(Name: sessions:
Store(hdfs://aws09.preprod.ubithere.com:9000/tmp/temp-1404953096/tmp819396740:org.apache.pig.impl.io.InterStorage)
- scope-93 Operator Key: scope-93)
2011-07-28 08:17:51,225 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler
- File concatenation threshold: 100 optimistic? false
2011-07-28 08:17:51,281 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2011-07-28 08:17:51,281 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2011-07-28 08:17:51,350 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 3
2011-07-28 08:17:51,350 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 3
2011-07-28 08:17:51,402 [main] INFO  
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
added to the job
2011-07-28 08:17:51,411 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to
default 0.3
2011-07-28 08:17:51,470 [main] INFO  org.apache.zookeeper.ZooKeeper
- Client environment:zookeeper.version=3.3.2-1031432, built on
11/05/2010 05:32 GMT
2011-07-28 08:17:51,470 [main] INF

I've reported the issue here:  
https://issues.apache.org/jira/browse/PIG-2193

Still investigating, but seems so far that the FILTER clause makes
the HBase loader loose all fields that are not explicitly used in
the script

I striped down the request to:

start_sessions = LOAD 'startSession.mde253811.preprod.ubithere.com'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
meta:infoid meta:imei meta:timestamp') AS (sid:chararray,
infoid:chararray, imei:chararray, start:long);
end_sessions = LOAD 'endSession.mde253811.preprod.ubithere.com'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
meta:timestamp meta:locid') AS (sid:chararray, end:long,
locid:chararray);
sessions = JOIN start_sessions BY sid, end_sessions BY sid;
sessions = FILTER sessions BY end > start AND end - start < 86400000L;
dump sessions;

and in the result, the fields "infoid", "imei" and "locid" are
empty, whereas the fields "sid", "start", "stop" are present.

(00000A2A33254B8FAE1E9AEAB2428EBE,,,1310649832970,00000A2A33254B8FAE1E9AEAB2428EBE,1310649838390,)
(00001DCECDC842C0A745C151B9EC295F,,,1310628836846,00001DCECDC842C0A745C151B9EC295F,1310628839075,)
(00001F8F2B3148D393963928188C72B6,,,1310681918742,00001F8F2B3148D393963928188C72B6,1310681949182,)
...

When using the HDFS loader, everything works correctly:

(00000A2A33254B8FAE1E9AEAB2428EBE,b87ac86bcf1d4cb44202aa826554a7b2,4e77d62e1839a470ec8386d42b85a076,1310649832970,00000A2A33254B8FAE1E9AEAB2428EBE,1310649838390,)
(00001DCECDC842C0A745C151B9EC295F,4a4bb0fff26e368c8209f1e480fdf70b,db3924d2e4b88bd103fa19aaa30a9af4,1310628836846,00001DCECDC842C0A745C151B9EC295F,1310628839075,)
(00001F8F2B3148D393963928188C72B6,5d7e58f68366b55d55862815f863a996,79e1ba90aa555e3e1041df4be657a11d,1310681918742,00001F8F2B3148D393963928188C72B6,1310681949182,)

...