Hi Jacob,

On May 13, 2012, at 2:03pm, Jacob Metcalf wrote:

Two comments…

1. Your pom.xml doesn't look like it's set up to build a proper Hadoop job jar.

After running "mvn assembly:assembly" you should have a job jar that has a lib subdirectory, and inside of that sub-dir you'll have all fo the jars (NOT the classes) for your dependent jars such as avro.

See http://exported.wordpress.com/2010/01/30/building-hadoop-job-jar-with-maven/

After running mvn assembly:assembly in your example directory I get a target/hadoop-example.jar file that's got Hadoop classes (and a bunch of others) all jammed inside it.

And your job jar shouldn't have Hadoop classes or jars inside it - those should be provided.

2. I would suggest using Hadoop 0.20.2 if you're on Cygwin.

That version avoids issues with Hadoop not being able to set permissions on local file system directories.


-- Ken

Ken Krugler
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr