java - Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf -

java - Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf -

i went through many questions regarding error couldn't find solution solve problem. here implementing sentiment analysis on twitter info using hadoop.

main class:

public class sentimentanalysis extends configured implements tool{ private static file file; public static class map extends mapreducebase implements mapper<longwritable, text, text, intwritable> { private final static intwritable 1 = new intwritable(1); private text word = new text(); classify classify = new classify(); /** * mapper reads tweets text file store * <"positive",1> or <"negative",1> */ public void map(longwritable key, text value, outputcollector<text, intwritable> output, reporter reporter) throws ioexception { string line = value.tostring();//streaming each tweet text file if (line != null) { word.set(classify.classify(line)); //invoke classify class tweet grouping of each text output.collect(word, one); } else { word.set("error"); output.collect(word, one);//key,value mapper } } } public static class cut down extends mapreducebase implements reducer<text, intwritable, text, intwritable> { /** * count frequency of each classified text grouping */ @override public void reduce(text key, iterator<intwritable> classifiedtext, outputcollector<text, intwritable> output, reporter reporter) throws ioexception { int sum = 0; while (classifiedtext.hasnext()) { sum += classifiedtext.next().get(); //sum frequency } output.collect(key, new intwritable(sum)); } } public static class classify { string[] categories; @suppresswarnings("rawtypes") lmclassifier lmc; /** * constructor loading serialized object created model class local * lmclassifer of class */ @suppresswarnings("rawtypes") public classify() { seek { lmc = (lmclassifier) abstractexternalizable.readobject(file); categories = lmc.categories(); } grab (classnotfoundexception e) { e.printstacktrace(); } grab (ioexception e) { e.printstacktrace(); } } /** * classify whether text positive or negative based on model object * * @param text * @return classified grouping i.e either positive or negative */ public string classify(string text) { conditionalclassification classification = lmc.classify(text); homecoming classification.bestcategory(); } } public static void main(string[] args) throws exception { int ret = toolrunner.run(new sentimentanalysis(), args); system.exit(ret); } @override public int run(string[] args) throws exception { if(args.length < 2) { system.out.println("invalid input , output directories"); homecoming -1; } jobconf conf = new jobconf(getconf(), sentimentanalysis.class); conf.setjobname("sentimentanalysis"); conf.setjarbyclass(sentimentanalysis.class); conf.setoutputkeyclass(text.class); conf.setoutputvalueclass(intwritable.class); conf.setmapoutputkeyclass(text.class); conf.setmapoutputvalueclass(intwritable.class); conf.setmapperclass(map.class); //conf.setcombinerclass(reduce.class); conf.setreducerclass(reduce.class); conf.setinputformat(textinputformat.class); conf.setoutputformat(textoutputformat.class); fileinputformat.setinputpaths(conf, new path(args[0])); fileoutputformat.setoutputpath(conf, new path(args[1])); file = new file(args[2]); jobclient.runjob(conf); homecoming 0; } }

error:

[cloudera@localhost ~]$ hadoop jar sentiment.jar sentimentanalysis test.txt sentimentoutput classifier.txt

test.txt contains few tweets sentiment need analysed. classifier.txt encoded text file helps classify (lmclassifier) class analyse tweets nowadays in test.txt.

14/10/05 20:59:23 warn mapred.jobclient: utilize genericoptionsparser parsing arguments. applications should implement tool same. 14/10/05 20:59:24 info mapred.fileinputformat: total input paths process : 1 14/10/05 20:59:24 info mapred.jobclient: running job: job_201410041909_0035 14/10/05 20:59:25 info mapred.jobclient: map 0% cut down 0% 14/10/05 20:59:41 info mapred.jobclient: task id : attempt_201410041909_0035_m_000000_0, status : failed java.lang.runtimeexception: error in configuring object @ org.apache.hadoop.util.reflectionutils.setjobconf(reflectionutils.java:109) @ org.apache.hadoop.util.reflectionutils.setconf(reflectionutils.java:75) @ org.apache.hadoop.util.reflectionutils.newinstance(reflectionutils.java:133) @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:413) @ org.apache.hadoop.mapred.maptask.run(maptask.java:332) @ org.apache.hadoop.mapred.child$4.run(child.java:268) @ java.security.accesscontroller.doprivileged(native method) @ javax.security.auth.subject.doas(subject.java:396) @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1438) @ org.apache.hadoop.mapred.child.main(child.java:262) caused by: java.lang.reflect.invocationtargetexception @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.ja 14/10/05 20:59:41 info mapred.jobclient: task id : attempt_201410041909_0035_m_000001_0, status : failed java.lang.runtimeexception: error in configuring object @ org.apache.hadoop.util.reflectionutils.setjobconf(reflectionutils.java:109) @ org.apache.hadoop.util.reflectionutils.setconf(reflectionutils.java:75) @ org.apache.hadoop.util.reflectionutils.newinstance(reflectionutils.java:133) @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:413) @ org.apache.hadoop.mapred.maptask.run(maptask.java:332) @ org.apache.hadoop.mapred.child$4.run(child.java:268) @ java.security.accesscontroller.doprivileged(native method) @ javax.security.auth.subject.doas(subject.java:396) @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1438) @ org.apache.hadoop.mapred.child.main(child.java:262) caused by: java.lang.reflect.invocationtargetexception @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.ja 14/10/05 20:59:52 info mapred.jobclient: task id : attempt_201410041909_0035_m_000000_1, status : failed java.lang.runtimeexception: error in configuring object @ org.apache.hadoop.util.reflectionutils.setjobconf(reflectionutils.java:109) @ org.apache.hadoop.util.reflectionutils.setconf(reflectionutils.java:75) @ org.apache.hadoop.util.reflectionutils.newinstance(reflectionutils.java:133) @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:413) @ org.apache.hadoop.mapred.maptask.run(maptask.java:332) @ org.apache.hadoop.mapred.child$4.run(child.java:268) @ java.security.accesscontroller.doprivileged(native method) @ javax.security.auth.subject.doas(subject.java:396) @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1438) @ org.apache.hadoop.mapred.child.main(child.java:262) caused by: java.lang.reflect.invocationtargetexception @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.ja 14/10/05 20:59:53 info mapred.jobclient: task id : attempt_201410041909_0035_m_000001_1, status : failed java.lang.runtimeexception: error in configuring object @ org.apache.hadoop.util.reflectionutils.setjobconf(reflectionutils.java:109) @ org.apache.hadoop.util.reflectionutils.setconf(reflectionutils.java:75) @ org.apache.hadoop.util.reflectionutils.newinstance(reflectionutils.java:133) @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:413) @ org.apache.hadoop.mapred.maptask.run(maptask.java:332) @ org.apache.hadoop.mapred.child$4.run(child.java:268) @ java.security.accesscontroller.doprivileged(native method) @ javax.security.auth.subject.doas(subject.java:396) @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1438) @ org.apache.hadoop.mapred.child.main(child.java:262) caused by: java.lang.reflect.invocationtargetexception @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.ja 14/10/05 21:00:04 info mapred.jobclient: task id : attempt_201410041909_0035_m_000000_2, status : failed java.lang.runtimeexception: error in configuring object @ org.apache.hadoop.util.reflectionutils.setjobconf(reflectionutils.java:109) @ org.apache.hadoop.util.reflectionutils.setconf(reflectionutils.java:75) @ org.apache.hadoop.util.reflectionutils.newinstance(reflectionutils.java:133) @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:413) @ org.apache.hadoop.mapred.maptask.run(maptask.java:332) @ org.apache.hadoop.mapred.child$4.run(child.java:268) @ java.security.accesscontroller.doprivileged(native method) @ javax.security.auth.subject.doas(subject.java:396) @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1438) @ org.apache.hadoop.mapred.child.main(child.java:262) caused by: java.lang.reflect.invocationtargetexception @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.ja 14/10/05 21:00:04 info mapred.jobclient: task id : attempt_201410041909_0035_m_000001_2, status : failed java.lang.runtimeexception: error in configuring object @ org.apache.hadoop.util.reflectionutils.setjobconf(reflectionutils.java:109) @ org.apache.hadoop.util.reflectionutils.setconf(reflectionutils.java:75) @ org.apache.hadoop.util.reflectionutils.newinstance(reflectionutils.java:133) @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:413) @ org.apache.hadoop.mapred.maptask.run(maptask.java:332) @ org.apache.hadoop.mapred.child$4.run(child.java:268) @ java.security.accesscontroller.doprivileged(native method) @ javax.security.auth.subject.doas(subject.java:396) @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1438) @ org.apache.hadoop.mapred.child.main(child.java:262) caused by: java.lang.reflect.invocationtargetexception @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.ja 14/10/05 21:00:19 info mapred.jobclient: job complete: job_201410041909_0035 14/10/05 21:00:19 info mapred.jobclient: counters: 7 14/10/05 21:00:19 info mapred.jobclient: job counters 14/10/05 21:00:19 info mapred.jobclient: failed map tasks=1 14/10/05 21:00:19 info mapred.jobclient: launched map tasks=8 14/10/05 21:00:19 info mapred.jobclient: data-local map tasks=8 14/10/05 21:00:19 info mapred.jobclient: total time spent maps in occupied slots (ms)=98236 14/10/05 21:00:19 info mapred.jobclient: total time spent reduces in occupied slots (ms)=0 14/10/05 21:00:19 info mapred.jobclient: total time spent maps waiting after reserving slots (ms)=0 14/10/05 21:00:19 info mapred.jobclient: total time spent reduces waiting after reserving slots (ms)=0 14/10/05 21:00:19 info mapred.jobclient: job failed: na exception in thread "main" java.io.ioexception: job failed! @ org.apache.hadoop.mapred.jobclient.runjob(jobclient.java:1416) @ sentimentanalysis.run(sentimentanalysis.java:124) @ org.apache.hadoop.util.toolrunner.run(toolrunner.java:70) @ org.apache.hadoop.util.toolrunner.run(toolrunner.java:84) @ sentimentanalysis.main(sentimentanalysis.java:101) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:25) @ java.lang.reflect.method.invoke(method.java:597) @ org.apache.hadoop.util.runjar.main(runjar.java:208)

i have no experience using hadoop, if "look" @ stack trace appears runtime exception in org.apache.hadoop.util.reflectionutils.setjobconf...

private static void setjobconf(object theobject, configuration conf) { 75 //if jobconf , jobconfigurable in classpath, , 76 //theobject of type jobconfigurable , 77 //conf of type jobconf 78 //invoke configure on theobject 79 seek { 80 class<?> jobconfclass = 81 conf.getclassbyname("org.apache.hadoop.mapred.jobconf"); 82 class<?> jobconfigurableclass = 83 conf.getclassbyname("org.apache.hadoop.mapred.jobconfigurable"); 84 if (jobconfclass.isassignablefrom(conf.getclass()) && 85 jobconfigurableclass.isassignablefrom(theobject.getclass())) { 86 method configuremethod = 87 jobconfigurableclass.getmethod("configure", jobconfclass); 88 configuremethod.invoke(theobject, conf); 89 } 90 } grab (classnotfoundexception e) { 91 //jobconf/jobconfigurable not in classpath. no need configure 92 } grab (exception e) { 93 throw new runtimeexception("error in configuring object", e); 94 } 95 }

clearly, both jobconf , jobconfigurable classes in classpath (otherwise have fallen thru cnfe grab block) exception has occurred... appears though nested exception java.lang.reflect.invocationtargetexception suggests there problem 'invoke' @ line 88 above.

so, trying invoke 'configure' method on target job instance config passed in failing.

the invocationtargetexception has wrappered actual causal exception need somehow grab runtimeexception @ top level e.getcause().getcause().printstacktrace() find out why invocation of configure method failed.

java ubuntu hadoop mapreduce runtimeexception