Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo, mail # user - Problem with using iterators to access table


Copy link to this message
-
Re: Problem with using iterators to access table
Jim Klucar 2012-06-27, 19:55
Jonathan,

You have to deploy all the dependencies of your iterator into the
lib/ext also, except whatever is on the Accumulo classpath (accumulo,
hadoop, zookeeper, etc).  I seem to remember that the mega-jar with
all dependencies built into one file doesn't work. The classloader
won't search the lib path inside the jar for other jars and load them.

What's going on here is that the tserver process has a separate
classloader for jars in lib/ext. When it detects a change in the
directory, it dynamically reloads all the jars in the directory. This
is how "user" jars can get incorporated into the tablet server
iterator stack without restarting the tservers.

On Wed, Jun 27, 2012 at 3:48 PM, Jonathan Hsu <[EMAIL PROTECTED]> wrote:
> Thanks for your quick reply.
>
> I didn't deploy my iterator jar to the directory, so I did that and tried to
> run it again.  I looked in the .debug.log file, and now I'm getting
> NoClassDefFoundErrors for classes that are not in the same package.  For
> example, I imported and am using the imported Jama Matrix class, and it
> threw a ClassNotFoundException on that class.  Am I missing something
> important with regard to importing external classes?
>
> Thanks again,
>
> Jonathan Hsu
>
>
> On Wed, Jun 27, 2012 at 1:16 PM, Jim Klucar <[EMAIL PROTECTED]> wrote:
>>
>> My first thought would be that you didn't deploy your iterator jar to
>> the accumulo lib/ext directory. That wouldn't be causing the
>> "built-in" iterators to produce an error also. Unfortunately this
>> error message isn't very informative because all the client knows is
>> that a Thrift service call somehow failed. The error message you want
>> to look at will be in the tserver_<hostname>.log file (or .debug.log
>> file) on the tserver. This is in $ACCUMULO_HOME/logs by default.
>>
>> On Wed, Jun 27, 2012 at 1:03 PM, Jonathan Hsu <[EMAIL PROTECTED]>
>> wrote:
>> > I'm trying to use iterators to access a table that I created.  When I
>> > run a
>> > MockInstance, the iterators work perfectly, but they don't work when I
>> > try
>> > to use them on an actual table.  Also, if I don't add the iterators, the
>> > scanner works fine as well.  I wrote both iterators, but the error
>> > doesn't
>> > seem to be with my iterators, because I also attempted the same thing
>> > with
>> > one of accumulo's iterators, and got the same result.  I know that the
>> > table
>> > is created correctly (by checking the accumulo shell).
>> >
>> > Here is my code :
>> >
>> > /* created the table first */
>> >
>> > Scanner scann = conn.createScanner(table, new Authorizations());
>> > scann.setRange(new Range("term1"));
>> >
>> > IteratorSetting cfg = new IteratorSetting(10, MyIterator1.class,
>> > properties);
>> > scann.addScanIterator(cfg);
>> >
>> > IteratorSetting cfg2 = new IteratorSetting(11, MyIterator2.class,
>> > properties);
>> > scann.addScanIterator(cfg2);
>> >
>> >
>> >
>> > I'm getting this error :
>> >
>> > Exception in thread "main" java.lang.RuntimeException:
>> > org.apache.accumulo.core.client.impl.AccumuloServerException: Error on
>> > server127.0.0.1:9997
>> > at
>> >
>> > org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>> > at
>> >
>> > com.bah.applefox.test.plugins.fulltextindex.DebugTest.main(DebugTest.java:151)
>> >
>> > Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException:
>> > Error on server 127.0.0.1:9997
>> > at
>> >
>> > org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>> > at
>> >
>> > org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>> > at
>> >
>> > org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>> > ... 1 more
>> >
>> > Caused by: org.apache.thrift.TApplicationException: Internal error
>> > processing startScan
>> > at
>> >
>> > org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>> > at
>> >
>> > org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)