Assignment 2 FAQ
How to set the number of reducers
To set the number of reducers to a certain number you can use the setNumReduceTask() of the Job class, and use the argument passed to the getPartition() method to distribute the received record. the default number of reducer is one reducer per node.
published on 09/06/2015 23:16:50 by Jumana Nassour-kassis
java.io.IOException: No FileSystem for scheme: hdfs
If you're receiving java.io.IOException: No FileSystem for scheme: hdfs then you need to add the following jar to your steps: http://central.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.2.0/hadoop-hdfs-2.2.0.jar
published on 22/05/2015 10:46:40 by Jumana Nassour-kassis
Caused by: java.lang.ClassNotFoundException: Class Step1$MapClass not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
Solution: When you create a runnable jar, choose: extract required libraries into generated jar.
published on 22/05/2015 10:45:24 by Jumana Nassour-kassis