Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> Unable to load Iterator with setscaniter and setshelliter


Copy link to this message
-
Re: Unable to load Iterator with setscaniter and setshelliter
general.vfs.classpaths is not set?
I saw in an earlier email that you are setting the context classloader. There are some places where the context manager was not being used when it should. This could be one of those cases. I would submit a jira ticket with all the information and I will try to look at it in the next day or so. Be sure to include classpath settings, list of files in HDFS for those locations, and the commands that are failing.
Sent via the Samsung GALAXY S®4, an AT&T 4G LTE smartphone

<div>-------- Original message --------</div><div>From: David Medinets <[EMAIL PROTECTED]> </div><div>Date:06/15/2014  10:53 AM  (GMT-05:00) </div><div>To: accumulo-user <[EMAIL PROTECTED]> </div><div>Subject: Re: Unable to load Iterator with setscaniter and setshelliter </div><div>
</div>The classpath settings in accumulo-site.xml are the following (which I think is the default):

    <property>
      <name>general.classpaths</name>
      <value>
    $ACCUMULO_HOME/server/target/classes/,
    $ACCUMULO_HOME/core/target/classes/,
    $ACCUMULO_HOME/start/target/classes/,
    $ACCUMULO_HOME/examples/target/classes/,
        $ACCUMULO_HOME/lib/[^.].$ACCUMULO_VERSION.jar,
        $ACCUMULO_HOME/lib/[^.].*.jar,
        $ZOOKEEPER_HOME/zookeeper[^.].*.jar,
        $HADOOP_HOME/conf,
        $HADOOP_HOME/[^.].*.jar,
        $HADOOP_HOME/lib/[^.].*.jar,
      </value>
      <description>Classpaths that accumulo checks for updates and class files.
      When using the Security Manager, please remove the ".../target/classes/" values.
      </description>
    </property>
On Sun, Jun 15, 2014 at 10:49 AM, dlmarion <[EMAIL PROTECTED]> wrote:
What does your classpath settings look like in accumulo-site.xml. I recently made some fixes in 1.6.1-Snapshot where the context classloader was not being used in all cases. I dont think this case was affected though.
Sent via the Samsung GALAXY S®4, an AT&T 4G LTE smartphone
From: Josh Elser
Date:06/15/2014 10:31 AM (GMT-05:00)
To: [EMAIL PROTECTED]
Subject: Re: Unable to load Iterator with setscaniter and setshelliter

Naw, the commons-vfs loader should be loading those resources using a second classloader.

Maybe it's just a problem with the HDFS code? Does it work if you have the jar with your iterator in lib/ or lib/ext? Or, maybe something is wrong like you defined a private constructor which is throwing that Exception?

On Jun 15, 2014 8:50 AM, "William Slacum" <[EMAIL PROTECTED]> wrote:
Wouldn't the iterator have to be on the classpath for the JVM that launches the shell command?
On Sun, Jun 15, 2014 at 9:02 AM, Vicky Kak <[EMAIL PROTECTED]> wrote:
The above line fails for me with the similar kind of error i.e ClassNotFoundException

root@accumulo atest> setiter -n MyIterator -p 10 -scan -minc -majc -class org.dallaybatta.MyIterator
2014-06-15 18:20:18,061 [shell.Shell] ERROR: org.apache.accumulo.shell.ShellCommandException: Command could not be initialized (Unable to load org.dallaybatta.MyIterator; class not found.)
My hdfs contains the corresponding jars but it yet fails.
After digging a code for a while I figured that the error is coming from org.apache.accumulo.shell.commands.SetIterCommand::execute

*********************************************************
    try {
      clazz = classloader.loadClass(className).asSubclass(SortedKeyValueIterator.class);
      untypedInstance = clazz.newInstance();
    } catch (ClassNotFoundException e) {
      StringBuilder msg = new StringBuilder("Unable to load ").append(className);
      if (className.indexOf('.') < 0) {
        msg.append("; did you use a fully qualified package name?");
      } else {
        msg.append("; class not found.");
      }
      throw new ShellCommandException(ErrorCode.INITIALIZATION_FAILURE, msg.toString());
    } catch (InstantiationException e) {

*********************************************************

Typically the ClassNotFoundException can appear also if the dependent classes are not present, here SortedKeyValueIterator could be the use case, I moved the accumulo core to the classpath folder in the hdfs but still could not get it working. May be some other dependent classes are required, needs more time to analyse in this direction.

The document states the following so it should ideally work,
"although the VFS classloader allows for classpath manipulation using a variety of schemes including URLs and HDFS URIs."

I find it strange that your test and my tests results differ as you are able to set Iterator for a single row but I am not.

Thanks,
Vicky
On Sun, Jun 15, 2014 at 8:25 AM, David Medinets <[EMAIL PROTECTED]> wrote:
I'm sure that I'm overlooking something simple. I can load my iterator using setiter but not with setscaniter or setshelliter.

Here is my do-nothing iterator:

public class MyIterator extends WrappingIterator implements OptionDescriber {

    @Override
    public IteratorOptions describeOptions() {
        String name = "dummy";
        String description = "Dummy Description";
        Map<String, String> namedOptions = new HashMap<String, String>();
        List<String> unnamedOptionDescriptions = null;
        return new IteratorOptions(name, description, namedOptions, unnamedOptionDescriptions);
    }

    @Override
    public boolean validateOptions(Map<String, String> options) {
        return true;
    }
    
}

I copy the jar file out to HDFS:

hadoop fs -mkdir /user/vagrant/d4m/classpath
hadoop fs -put /vagrant/schema/target/d4m_schema-0.0.1-SNAPSHOT.jar /user/vagrant/classpath

I set the table-specific classpath context:

createtable atest
table atest
insert row cf cq value
config -s general.vfs.context.classpath.d4m=hdfs://affy-master:9000/user/vagrant/classpath
config -t atest -s table.classpath.context=d4m

Now I can configure the iterator and scan over the single row without a problem:

setiter -n MyIterator -p 10 -scan -minc -majc -class com.codebits.d4m.iterator.MyIterator
scan