Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Loading data in different timezone


Copy link to this message
-
Loading data in different timezone
Hi All,

So we have data in S3 partitioned by hour in UTC :

2012/10/11/00
2012/10/11/01
....

2012/10/12/00
2012/10/12/01

We need to now load data in Pacific time so we need to load for 2012/10/12
data from

2012/10/11/08
2012/10/11/09
....

2012/10/12/00
.....
2012/10/12/07

We also need to load data for an entire month in Pacific.

So far what we have planned to do, is to extend TextLoader (we want to use
TextLoader). Send in the date or the month and then return a list of dates
to be loaded.

Are there any better ideas out there?

Thanks,
Meg