Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Bulk load from OSGi running client


Copy link to this message
-
Re: Bulk load from OSGi running client
I opened a ticket HBASE-9682<https://issues.apache.org/jira/browse/HBASE-9682>.
I will try to provide some sort of sample project.

Thanks.
On Fri, Sep 27, 2013 at 8:06 PM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:

> Hi Amit,
>
> Would you be able to open a ticket summarizing your findings? Can you
> provide a sample project that demonstrates the behavior you're seeing? We
> could use that to provide a fix and, I hope, some kind of unit or
> integration test.
>
> Thanks,
> Nick
>
>
> On Sun, Sep 22, 2013 at 6:10 AM, Amit Sela <[EMAIL PROTECTED]> wrote:
>
> > I think I got it.
> > Algorithm has a Configuration member that is instantiated when Algorithm
> is
> > created. Since Algorithm is static, when I update a jar that doesn't
> cause
> > the hbase bundle to update, configuration still holds to the old CL.
> > I tested with GZ and changed the code to call new Configuration (in
> > getCodec(Configuration conf) GZ override) and it works. Makes sense
> because
> > it probably takes the current TCCL.
> >
> > Since Algorithm.getCodec(Configuration conf) is only called from inside
> > Algorithm and the only (apparent) reason to have configuration as a
> member
> > of Algorithm is to avoid code duplication of
> > *conf.setBoolean("hadoop.native.lib",
> > true)* for every codec, I think it would be better to duplicate code here
> > even if it's not as pretty as it is now, because on top of cloning the
> > configuration properties, the clone constructor also clones the CL and
> that
> > may cause problems where CL changes like in OSGi environment.
> >
> > The NPE causing recovery is still a mystery to me :)
> >
> >
> > On Wed, Sep 11, 2013 at 3:34 PM, Amit Sela <[EMAIL PROTECTED]> wrote:
> >
> > > I did some more digging and I got this:
> > >
> > > When the HBase bundle is loaded (system start) the
> Compression.Algorithm
> > > is probably created for the first time and the constructor calls new
> > > Configuration().
> > > When I update a bundle (but not the HBase bundle) it refreshes the
> > > relevant packages, which doesn't include the HBase bundle, and once I
> try
> > > to use getCodec() (GZ in my case) it creates a
> > > new ReusableStreamGzipCodec() and sets new Configuration(*conf*) -
> where
> > *conf
> > > *is a private final member in Algorithm. Since *conf *holds the old
> class
> > > loader (referring to the pre-update bundle) it passes that CL to the
> new
> > > configuration created for the codec.
> > >
> > > I still have NO IDEA why NPE would cause Compression.Algorithm to
> > > re-instantiate itself...
> > > I think that calling new Configuration() and
> > > setting this.conf.setBoolean("hadoop.native.lib", true) for each codec
> > > would solve it since the class loader that would be set is the TCCL.
> > >
> > > I'll give it a try an keep updating.
> > >
> > > Thanks,
> > > Amit.
> > >
> > >
> > >
> > > On Mon, Sep 9, 2013 at 9:12 PM, Stack <[EMAIL PROTECTED]> wrote:
> > >
> > >> On Mon, Sep 9, 2013 at 12:14 AM, Amit Sela <[EMAIL PROTECTED]>
> wrote:
> > >> ...
> > >>
> > >> > The main issue still remains, it looks like Compression.Algortihm
> > >> > configuration's class loader had reference to the bundle in
> revision 0
> > >> > (before jar update) instead of revision 1 (after jar update). This
> > >> could be
> > >> > because of caching (or static) but then, why should it work after I
> > get
> > >> > NullPointerException (it does, immediately, no restarts or bundle
> > >> updates).
> > >> >
> > >>
> > >> When you say configuration above, you mean Compression.Algorithm's
> > >> reference to an Hadoop Configuration instance?  I've not looked at
> code.
> > >>  Is it coming in via static?
> > >>
> > >> I am not sure what it would then start working after NPE.  I would
> > expect
> > >> that it would stay broke rather than recover.
> > >>
> > >> St.Ack
> > >>
> > >
> > >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB