Hi,

I just updated to hadoop 0.18.1 and got a clean version of mahout from svn.
However, I am having problems with KMeans, that can be traced down to :

2008-10-25 19:10:16,987 INFO org.apache.hadoop.mapred.Merger: Merging
2 sorted segments
2008-10-25 19:10:16,987 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 2 segments left of total size: 5011 bytes
2008-10-25 19:10:16,999 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200810251826_0013_r_000000_0 Merge of the inmemory files threw
an exception: java.io.IOException: Intermedate merge failed
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2078)
Caused by: java.lang.NumberFormatException: For input string: "["
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1224)
at java.lang.Double.parseDouble(Double.java:510)
at org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
at org.apache.mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
at org.apache.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:38)
at org.apache.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:31)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.combineAndSpill(ReduceTask.java:2174)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access$3100(ReduceTask.java:341)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
... 1 more

2008-10-25 19:10:16,999 INFO org.apache.hadoop.mapred.ReduceTask:
In-memory merge complete: 0 files left.
2008-10-25 19:10:17,000 WARN org.apache.hadoop.mapred.TaskTracker:
Error running child
java.io.IOException: attempt_200810251826_0013_r_000000_0The reduce
copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
This is while running the synthetic_control.data example, but I have the
same problems with any other input data.

I am able to do other map-reduce job without problems.

Here is the output of the jar task:

hadoop@philippe-vaio:/usr/local/hadoop$ bin/hadoop jar
/home/philippe/workspace/MahoutJava/examples/dist/apache-mahout-examples-0.1-dev.jar
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
08/10/25 19:09:27 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/25 19:09:28 INFO mapred.FileInputFormat: Total input paths to process
: 1
08/10/25 19:09:28 INFO mapred.FileInputFormat: Total input paths to process
: 1
08/10/25 19:09:28 INFO mapred.JobClient: Running job: job_200810251826_0010
08/10/25 19:09:29 INFO mapred.JobClient:  map 0% reduce 0%
08/10/25 19:09:31 INFO mapred.JobClient:  map 50% reduce 0%
08/10/25 19:09:32 INFO mapred.JobClient: Job complete: job_200810251826_0010
08/10/25 19:09:32 INFO mapred.JobClient: Counters: 7
08/10/25 19:09:32 INFO mapred.JobClient:   File Systems
08/10/25 19:09:32 INFO mapred.JobClient:     HDFS bytes read=291644
08/10/25 19:09:32 INFO mapred.JobClient:     HDFS bytes written=323660
08/10/25 19:09:32 INFO mapred.JobClient:   Job Counters
08/10/25 19:09:32 INFO mapred.JobClient:     Launched map tasks=2
08/10/25 19:09:32 INFO mapred.JobClient:     Data-local map tasks=2
08/10/25 19:09:32 INFO mapred.JobClient:   Map-Reduce Framework
08/10/25 19:09:32 INFO mapred.JobClient:     Map input records=600
08/10/25 19:09:32 INFO mapred.JobClient:     Map input bytes=288374
08/10/25 19:09:32 INFO mapred.JobClient:     Map output records=600
08/10/25 19:09:32 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/25 19:09:32 INFO mapred.FileInputFormat: Total input paths to process
: 2
08/10/25 19:09:32 INFO mapred.FileInputFormat: Total input paths to process
08/10/25 19:09:32 INFO mapred.JobClient: Running job: job_200810251826_0011
08/10/25 19:09:33 INFO mapred.JobClient:  map 0% reduce 0%
08/10/25 19:09:37 INFO mapred.JobClient:  map 50% reduce 0%
08/10/25 19:09:39 INFO mapred.JobClient:  map 100% reduce 0%
08/10/25 19:09:44 INFO mapred.JobClient:  map 100% reduce 16%
08/10/25 19:09:52 INFO mapred.JobClient: Job complete: job_200810251826_0011
08/10/25 19:09:52 INFO mapred.JobClient: Counters: 16
08/10/25 19:09:52 INFO mapred.JobClient:   File Systems
08/10/25 19:09:52 INFO mapred.JobClient:     HDFS bytes read=323660
08/10/25 19:09:52 INFO mapred.JobClient:     HDFS bytes written=1447
08/10/25 19:09:52 INFO mapred.JobClient:     Local bytes read=1389
08/10/25 19:09:52 INFO mapred.JobClient:     Local bytes written=37878
08/10/25 19:09:52 INFO mapred.JobClient:   Job Counters
08/10/25 19:09:52 INFO mapred.JobClient:     Launched reduce tasks=1
08/10/25 19:09:52 INFO mapred.JobClient:     Launched map tasks=2
08/10/25 19:09:52 INFO mapred.JobClient:     Data-local map tasks=2
08/10/25 19:09:52 INFO mapred.JobClient:   Map-Reduce Framework
08/10/25 19:09:52 INFO mapred.JobClient:     Reduce input groups=1
08/10/25 19:09:52 INFO mapred.JobClient:     Combine output records=29
08/10/25 19:09:52 INFO mapred.JobClient:     Map input records=600
08/10/25 19:09:52 INFO mapred.JobClient:     Reduce output records=1
08/10/25 19:09:52 INFO mapred.JobClient:     Map output bytes=943020
08/10/25 19:09:52 INFO mapred.JobClient:     Map input bytes=323660
08/10/25 19:09:52 INFO mapred.JobClient:     Combine input records=1760
08/10/25 19:09:52 INFO mapred.JobClient:     Map output records=1732
08/10/25 19:09:52 INFO mapred.JobClient:     Reduce input records=1
08/10/25 19:09:53 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/25 19:09:53 INFO mapred.FileInputFormat: Total input paths to process
08/10/25 19:09:53 INFO mapred.FileInputFormat: Total input

Did this work with 0.18.0 or other prior versions for you?
On Oct 25, 2008, at 7:23 PM, Philippe Lamarche wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Unfortunately, I went straight from 0.17.2 to 0.18.1.  It was working on
0.17.2.

On Sun, Oct 26, 2008 at 9:48 AM, Grant Ingersoll <[EMAIL PROTECTED]>wrote:

Same Mahout code, though, right?

Can you provide details on how you were running it?

On Oct 26, 2008, at 10:46 AM, Philippe Lamarche wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Hi,

My goal is to run the example KMeans. I must download the synthetic control
data and put it on the dfs in "testdata".

To be sure that everything is ok, I stated form a clean state on my laptop.

I downloaded hadoop 0.18.1.

I changed the conf/hadoop-site.xml to this:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/hadoop-datastore/hadoop-${user.name}</value>
  </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>hdfs://localhost:9001</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

I changed JAVA_HOME in hadoop-env.sh.

I downloaded mahout from SVN, at revision 708282.

I built both core and example from ant script.

I copied apache-mahout-core-0.1-dev.jar to {hadoop-home}/lib.

I downloaded
http://archive.ics.uci.edu/ml/databases/synthetic_control/synthetic_control.data

I added the file to the dfs:
hadoop@phil:/usr/local/hadoop$ bin/hadoop dfs -put
/home/philippe/synthetic_control.data testdata

I ran the example jar, but it failed :

hadoop@phil:/usr/local/hadoop$ bin/hadoop jar
/home/philippe/workspace/MahoutJava/examples/dist/apache-mahout-examples-0.1-dev.jar
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
08/10/27 15:34:55 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/27 15:34:55 INFO mapred.FileInputFormat: Total input paths to process
: 1
08/10/27 15:34:55 INFO mapred.FileInputFormat: Total input paths to process
: 1
08/10/27 15:34:55 INFO mapred.JobClient: Running job: job_200810271532_0001
08/10/27 15:34:56 INFO mapred.JobClient:  map 0% reduce 0%
08/10/27 15:34:59 INFO mapred.JobClient: Job complete: job_200810271532_0001
08/10/27 15:34:59 INFO mapred.JobClient: Counters: 7
08/10/27 15:34:59 INFO mapred.JobClient:   File Systems
08/10/27 15:34:59 INFO mapred.JobClient:     HDFS bytes read=291644
08/10/27 15:34:59 INFO mapred.JobClient:     HDFS bytes written=323660
08/10/27 15:34:59 INFO mapred.JobClient:   Job Counters
08/10/27 15:34:59 INFO mapred.JobClient:     Launched map tasks=2
08/10/27 15:34:59 INFO mapred.JobClient:     Data-local map tasks=2
08/10/27 15:34:59 INFO mapred.JobClient:   Map-Reduce Framework
08/10/27 15:34:59 INFO mapred.JobClient:     Map input records=600
08/10/27 15:34:59 INFO mapred.JobClient:     Map input bytes=288374
08/10/27 15:34:59 INFO mapred.JobClient:     Map output records=600
08/10/27 15:34:59 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/27 15:35:00 INFO mapred.FileInputFormat: Total input paths to process
: 2
08/10/27 15:35:00 INFO mapred.FileInputFormat: Total input paths to process
: 2
08/10/27 15:35:00 INFO mapred.JobClient: Running job: job_200810271532_0002
08/10/27 15:35:01 INFO mapred.JobClient:  map 0% reduce 0%
08/10/27 15:35:10 INFO mapred.JobClient:  map 100% reduce 0%
08/10/27 15:35:16 INFO mapred.JobClient: Job complete: job_200810271532_0002
08/10/27 15:35:16 INFO mapred.JobClient: Counters: 16
08/10/27 15:35:16 INFO mapred.JobClient:   File Systems
08/10/27 15:35:16 INFO mapred.JobClient:     HDFS bytes read=323660
08/10/27 15:35:16 INFO mapred.JobClient:     HDFS bytes written=1447
08/10/27 15:35:16 INFO mapred.JobClient:     Local bytes read=1389
08/10/27 15:35:16 INFO mapred.JobClient:     Local bytes written=37878
08/10/27 15:35:16 INFO mapred.JobClient:   Job Counters
08/10/27 15:35:16 INFO mapred.JobClient:     Launched reduce tasks=1
08/10/27 15:35:16 INFO mapred.JobClient:     Launched map tasks=2
08/10/27 15:35:16 INFO mapred.JobClient:     Data-local map tasks=2
08/10/27 15:35:16 INFO mapred.JobClient:   Map-Reduce Framework
08/10/27 15:35:16 INFO mapred.JobClient:     Reduce input groups=1
08/10/27 15:35:16 INFO mapred.JobClient:     Combine output records=29
08/10/27 15:35:16 INFO mapred.JobClient:     Map input records=600
08/10/27 15:35:16 INFO mapred.JobClient:     Reduce output records=1
08/10/27 15:35:16 INFO mapred.JobClient:     Map output bytes=943020
08/10/27 15:35:16 INFO mapred.JobClient:     Map input bytes=323660
08/10/27 15:35:16 INFO mapred.JobClient:     Combine input records=1760
08/10/27 15:35:16 INFO mapred.JobClient:     Map output records=1732
08/10/27 15:35:16 INFO mapred.JobClient:     Reduce input records=1
08/10/27 15:35:16 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/27 15:35:16 INFO mapred.FileInputFormat: Total input paths to process
08/10/27 15:35:16 INFO mapred.FileInputFormat: Total input paths to process
08/10/27 15:35:16 INFO mapred.JobClient: Running job: job_200810271532_0003
08/10/27 15:35:17 INFO mapred.JobClient:  map 0% reduce 0%
08/10/27 15:35:24 INFO mapred.JobClient:  map 100% reduce 0%
08/10/27 15:35:28 INFO mapred.JobClient: Job complete: job_200810271532_0003
08/10/27 15:35:28 INFO mapred.JobClient: Counters: 16
08/10/27 15:35:28 INFO mapred.JobClient:   File Systems
08/10/27 15:35:28 INFO mapred.JobClient:     HDFS bytes read=326554
08/10/27 15:35:28 INFO mapred.JobClient:     HDFS bytes written=1137260
08/10/27 15:35:28 INFO mapred.JobClient:     Local bytes read=1147358
08/10/27 15:35:28 INFO mapred.JobClient:     Local bytes written=2304490
08/10/27 15:35:28 INFO mapred.JobClient:   Job Counters
08/10/27 15:35:28 INFO mapred.JobClient:     Launched reduce tasks=1
08/10/27 15:35:28 INFO mapred.JobClient:     Launched map tasks=2
08/10/27 15:35:28 INFO mapred.JobClient:     Data-local map tasks=2
08/10/27 15:35:28 INFO mapred.JobClient:   Map-Reduce Framework
08/10/27 15:35:28 INFO mapred.JobClient:     Reduce input groups=1
08/10/27 15:3

On Oct 27, 2008, at 4:26 PM, Philippe Lamarche wrote:

What happens if you don't do this but use the "job" file instead (ant  
job in the examples dir)?  I'm trying to replicate this, but am stuck  
at the moment.

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

I removed the apache-mahout-core-0.1-dev.jar file from {hadoop-home}/lib and
added apache-mahout-examples-0.1-dev.job

my lib folder now contain :
-rw-r--r-- 1 hadoop hadoop 4506592 2008-10-27 19:59
apache-mahout-examples-0.1-dev.job
-rw-r--r-- 1 hadoop root    258337 2008-10-27 14:37
commons-cli-2.0-SNAPSHOT.jar
-rw-r--r-- 1 hadoop root     46725 2008-10-27 14:37 commons-codec-1.3.jar
-rw-r--r-- 1 hadoop root    279781 2008-10-27 14:37
commons-httpclient-3.0.1.jar
-rw-r--r-- 1 hadoop root     38015 2008-10-27 14:37
commons-logging-1.0.4.jar
-rw-r--r-- 1 hadoop root     26202 2008-10-27 14:37
commons-logging-api-1.0.4.jar
-rw-r--r-- 1 hadoop root    180792 2008-10-27 14:37 commons-net-1.4.1.jar
-rw-r--r-- 1 hadoop root    288534 2008-10-27 14:37 jets3t-0.6.0.jar
-rw-r--r-- 1 hadoop root    665638 2008-10-27 14:37 jetty-5.1.4.jar
-rw-r--r-- 1 hadoop root     11358 2008-10-27 14:37 jetty-5.1.4.LICENSE.txt
drwxr-xr-x 2 hadoop root      4096 2008-10-27 14:37 jetty-ext
-rw-r--r-- 1 hadoop root    121070 2008-10-27 14:37 junit-3.8.1.jar
-rw-r--r-- 1 hadoop root     14999 2008-10-27 14:37 junit-3.8.1.LICENSE.txt
-rw-r--r-- 1 hadoop root      9484 2008-10-27 14:37 kfs-0.1.3.jar
-rw-r--r-- 1 hadoop root     11358 2008-10-27 14:37 kfs-0.1.LICENSE.txt
-rw-r--r-- 1 hadoop root    391834 2008-10-27 14:37 log4j-1.2.15.jar
drwxr-xr-x 4 hadoop root      4096 2008-10-27 14:37 native
-rw-r--r-- 1 hadoop root     65261 2008-10-27 14:37 oro-2.0.8.jar
-rw-r--r-- 1 hadoop root     97689 2008-10-27 14:37 servlet-api.jar
-rw-r--r-- 1 hadoop root     15345 2008-10-27 14:37 slf4j-api-1.4.3.jar
-rw-r--r-- 1 hadoop root      1159 2008-10-27 14:37 slf4j-LICENSE.txt
-rw-r--r-- 1 hadoop root      8601 2008-10-27 14:37 slf4j-log4j12-1.4.3.jar
-rw-r--r-- 1 hadoop root     15010 2008-10-27 14:37 xmlenc-0.52.jar

when I try to run the synthetic example I get:

hadoop@phil:/usr/local/hadoop$ bin/hadoop jar
/home/philippe/workspace/MahoutJava/examples/dist/apache-mahout-examples-0.1-dev.jar
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
java.lang.NoClassDefFoundError: org/apache/mahout/matrix/Vector
    at
org.apache.mahout.clustering.syntheticcontrol.canopy.InputDriver.runJob(InputDriver.java:42)
    at
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job.runJob(Job.java:77)
    at
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job.main(Job.java:44)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
    at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.matrix.Vector
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
    ... 12 more

Right now, Hadoop doesn't have any additional classpath elements that I know
of, from conf/hadoop-env.sh or elsewhere.

Did I understand correctly what you were saying?

On Mon, Oct 27, 2008 at 7:29 PM, Grant Ingersoll <[EMAIL PROTECTED]>wrote:

On Oct 27, 2008, at 8:14 PM, Philippe Lamarche wrote:

This should be job, not jar.  I'm trying to reproduce right now.
Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Ho,

Sorry about that, I didn't know what a job file was, and it answers my
questions about using the lib folders on a multiples nodes cluster!

However, I am still getting the same error.

hadoop@phil:/usr/local/hadoop$ bin/hadoop jar
/home/philippe/workspace/MahoutJava/examples/build/apache-mahout-examples-0.1-dev.job
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
08/10/27 21:01:36 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/27 21:01:37 INFO mapred.FileInputFormat: Total input paths to process
: 1
08/10/27 21:01:37 INFO mapred.FileInputFormat: Total input paths to process
: 1
08/10/27 21:01:38 INFO mapred.JobClient: Running job: job_200810272000_0001
08/10/27 21:01:39 INFO mapred.JobClient:  map 0% reduce 0%
08/10/27 21:01:46 INFO mapred.JobClient: Job complete: job_200810272000_0001
08/10/27 21:01:46 INFO mapred.JobClient: Counters: 7
08/10/27 21:01:46 INFO mapred.JobClient:   File Systems
08/10/27 21:01:46 INFO mapred.JobClient:     HDFS bytes read=291644
08/10/27 21:01:46 INFO mapred.JobClient:     HDFS bytes written=323660
08/10/27 21:01:46 INFO mapred.JobClient:   Job Counters
08/10/27 21:01:46 INFO mapred.JobClient:     Launched map tasks=2
08/10/27 21:01:46 INFO mapred.JobClient:     Data-local map tasks=2
08/10/27 21:01:46 INFO mapred.JobClient:   Map-Reduce Framework
08/10/27 21:01:46 INFO mapred.JobClient:     Map input records=600
08/10/27 21:01:46 INFO mapred.JobClient:     Map input bytes=288374
08/10/27 21:01:46 INFO mapred.JobClient:     Map output records=600
08/10/27 21:01:46 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/27 21:01:46 INFO mapred.FileInputFormat: Total input paths to process
: 2
08/10/27 21:01:46 INFO mapred.FileInputFormat: Total input paths to process
: 2
08/10/27 21:01:46 INFO mapred.JobClient: Running job: job_200810272000_0002
08/10/27 21:01:47 INFO mapred.JobClient:  map 0% reduce 0%
08/10/27 21:01:57 INFO mapred.JobClient:  map 100% reduce 0%
08/10/27 21:02:12 INFO mapred.JobClient: Job complete: job_200810272000_0002
08/10/27 21:02:12 INFO mapred.JobClient: Counters: 16
08/10/27 21:02:12 INFO mapred.JobClient:   File Systems
08/10/27 21:02:12 INFO mapred.JobClient:     HDFS bytes read=323660
08/10/27 21:02:12 INFO mapred.JobClient:     HDFS bytes written=1450
08/10/27 21:02:12 INFO mapred.JobClient:     Local bytes read=1392
08/10/27 21:02:12 INFO mapred.JobClient:     Local bytes written=37881
08/10/27 21:02:12 INFO mapred.JobClient:   Job Counters
08/10/27 21:02:12 INFO mapred.JobClient:     Launched reduce tasks=1
08/10/27 21:02:12 INFO mapred.JobClient:     Launched map tasks=2
08/10/27 21:02:12 INFO mapred.JobClient:     Data-local map tasks=2
08/10/27 21:02:12 INFO mapred.JobClient:   Map-Reduce Framework
08/10/27 21:02:12 INFO mapred.JobClient:     Reduce input groups=1
08/10/27 21:02:12 INFO mapred.JobClient:     Combine output records=29
08/10/27 21:02:12 INFO mapred.JobClient:     Map input records=600
08/10/27 21:02:12 INFO mapred.JobClient:     Reduce output records=1
08/10/27 21:02:12 INFO mapred.JobClient:     Map output bytes=943020
08/10/27 21:02:12 INFO mapred.JobClient:     Map input bytes=323660
08/10/27 21:02:12 INFO mapred.JobClient:     Combine input records=1760
08/10/27 21:02:12 INFO mapred.JobClient:     Map output records=1732
08/10/27 21:02:12 INFO mapred.JobClient:     Reduce input records=1
08/10/27 21:02:12 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/27 21:02:12 INFO mapred.FileInputFormat: Total input paths to process
: 2
08/10/27 21:02:12 INFO mapred.FileInputFormat: Total input paths to process
: 2
08/10/27 21:02:12 INFO mapred.JobClient: Running job: job_200810272000_0003
08/10/27 21:02:13 INFO mapred.JobClient:  map 0% reduce 0%
08/10/27 21:02:19 INFO mapred.JobClient:  map 100% reduce 0%
08/10/27 21:02:29 INFO mapred.JobClient: Job complete: job_200810272000_0003
08/10/27 21:02:29 INFO mapred.JobClient: Counters: 16
08/10/27 21:02:29 INFO mapred.JobClient:   File Systems
08/10/27 21:02:29 INFO mapred.JobClient:     HDFS bytes read=326560
08/10/27 21:02:29 INFO mapred.JobClient:     HDFS bytes written=1139060
08/10/27 21:02:29 INFO mapred.JobClient:     Local bytes read=1149158
08/10/27 21:02:29 INFO mapred.JobClient:     Local bytes written=2308118
08/10/27 21:02:29 INFO mapred.JobClient:   Job Counters
08/10/27 21:02:29 INFO mapred.JobClient:     Launched reduce tasks=1
08/10/27 21:02:29 INFO mapred.JobClient:     Launched map tasks=2
08/10/27 21:02:29 INFO mapred.JobClient:     Data-local map tasks=2
08/10/27 21:02:29 INFO mapred.JobClient:   Map-Reduce Framework
08/10/27 21:02:29 INFO mapred.JobClient:     Reduce input groups=1
08/10/27 21:02:29 INFO mapred.JobClient:     Combine output records=0
08/10/27 21:02:29 INFO mapred.JobClient:     Map input records=600
08/10/27 21:02:29 INFO mapred.JobClient:     Reduce output records=600
08/10/27 21:02:29 INFO mapred.JobClient:     Map output bytes=1141460
08/10/27 21:02:29 INFO mapred.JobClient:     Map input bytes=323660
08/10/27 21:02:29 INFO mapred.JobClient:     Combine input records=0
08/10/27 21:02:29 INFO mapred.JobClient:     Map output records=600
08/10/27 21:02:29 INFO mapred.JobClient:     Reduce input records=600
08/10/27 21:02:29 INFO kmeans.KMeansDriver: Iteration 0
08/10/27 21:02:29 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/27 21:02:30 INFO mapred.FileInputFormat: Total input paths to process
08/10/27 21:02:30 INFO mapred.FileInputFormat: Total input paths to process
08/10/27 21:02:30 INFO mapred.JobClient: Running job: job_200810272000_0004
08/10/27 21:02:31 INFO mapred.JobClient:  map 0% reduce 0%
08/10/27 21:02:39 INFO mapred.JobClient:  map 100% reduce 0%
08/10/27 21:02:46 INFO mapred.JobClient: T

OK, I can reproduce this.

On Oct 27, 2008, at 8:14 PM, Philippe Lamarche wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

That is, I can reproduce the original problem.

On Oct 27, 2008, at 9:22 PM, Grant Ingersoll wrote:
Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

On Oct 26, 2008, at 10:46 AM, Philippe Lamarche wrote:

BTW, are you saying the same exact code was working on 0.17.2 or are  
you referring to some older Mahout code that worked on 17.2?
Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

No,

I am doing some work with text clustering. I was struggling with memory
problems on hadoop 0.17.2 and an unknown version of Mahout. I usually update
mahout often but i could not say when or the exact version. I decided to
update all my things and my memory problems where transformed into this.

I feel that I updated Hadoop first, got the error and then, while trying to
resolve it, updated Mahout. There is only one way to find out: I'll try it
out on hadoop 0.17.2.

On Mon, Oct 27, 2008 at 9:33 PM, Grant Ingersoll <[EMAIL PROTECTED]>wrote:

OK, I can confirm that the exact same code works with 0.17.2 and not  
w/ 0.18.1.  So, it sounds like a bug in Hadoop, or we are relying on  
incorrect behavior in Hadoop.
On Oct 27, 2008, at 9:33 PM, Grant Ingersoll wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

I started a thread on core-user@hadoop.a.o:  http://hadoop.markmail.org/message/cczunzfhpcqz6pis
On Oct 27, 2008, at 9:49 PM, Grant Ingersoll wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Hi,

Are you guys running on real Hadoop arrays? I can run the synthetic
control example just fine on a single machine. That code is just trying
to read a vector from a string. I'd be surprised if we were using any
"features" but will watch the threads.

Jeff

Grant Ingersoll wrote:;

Just a single machine.  I didn't think we were using features either.  
Are you saying you can run the example using 0.18.1?

BTW, Philippe, what JVM, O/S, etc. are you using?

-Grant

On Oct 27, 2008, at 11:55 PM, Jeff Eastman wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Ubuntu linux 2.6.24 <http://2.6.24.21>, with java-6-sun-1.6.0.07.

On Tue, Oct 28, 2008 at 7:03 AM, Grant Ingersoll <[EMAIL PROTECTED]>wrote:

Hmm, OK, I'm on OS X (10.5.5):
java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06-153)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_07-b06-57, mixed mode)
-Grant

On Oct 28, 2008, at 7:19 AM, Philippe Lamarche wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Can you publish the steps you did to run?

I pretty much followed the Hadoop quick start, and just substituted in  
the Mahout Job for the grep job.

-Grant

On Oct 27, 2008, at 11:55 PM, Jeff Eastman wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

I'm also running Eclipse on OS X under Java 1.6. I can run all 3 of the
clustering examples without problems. Of course, I developed them in
this environment. Could this be a platform issue?

Jeff
Grant Ingersoll wrote:

I run the job classes from Eclipse, as a java application.
Jeff
Grant Ingersoll wrote:;

How, exactly are you doing it?  That might give some clues.

On Oct 28, 2008, at 12:06 PM, Jeff Eastman wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

I right-click on the syntheticcontrol.kmeans.Job.java file in the
Eclipse Package Explorer and select Run As... Java Application. My
Mahout project does not depend upon any other projects and uses the
hadoop-0.18.1 jar from our lib. It has always worked just fine. What are
you/he doing differently?

Jeff
Grant Ingersoll wrote:

We're doing pseudo-distributed: http://hadoop.apache.org/core/docs/r0.17.2/quickstart.html
  (further down)

Whereas you're doing, I think, Standalone.

On Oct 28, 2008, at 3:21 PM, Jeff Eastman wrote:
Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Philippe, can you try the patch suggested by Arun Murthy on core-user@hadoop.a.o
?  See http://issues.apache.org/jira/browse/HADOOP-4277

I'm pretty swamped at the moment w/ ApacheCon coming up next week, but  
if it does fix the issue, then maybe we should move forward to the  
18.2 candidate (I don't think it has been released yet, those guys  
have a pretty sophisticated build process going)

-Grant

On Oct 28, 2008, at 7:19 AM, Philippe Lamarche wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

I will!

On 10/29/08, Grant Ingersoll <[EMAIL PROTECTED]> wrote:;

I am not sure I understand the hadoop svn structure, however I was able to
make it work with hadoop trunk, or 0.20.0-dev.
It didn't work with hadoop/branch-0.18, with or without patch 4277.
Here is a copy-paste of the steps, once Hadoop is built and installed.  I am
using the same exact "apache-mahout-examples-0.1-dev.job", not rebuilt with
the 0.20.0-dev jars.

It works!

That would mean that the bug/feature is not related to
HADOOP-4277<http://issues.apache.org/jira/browse/HADOOP-4277>,
and was reintroduced (or never took away) in hadoop/trunk.
hadoop@phil:/usr/local/hadoop$ bin/hadoop namenode -format
08/10/29 18:27:59 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = phil/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.0-dev
STARTUP_MSG:   build =  -r ; compiled by 'philippe' on Wed Oct 29 18:25:08
EDT 2008
************************************************************/
08/10/29 18:28:00 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop
08/10/29 18:28:00 INFO namenode.FSNamesystem: supergroup=supergroup
08/10/29 18:28:00 INFO namenode.FSNamesystem: isPermissionEnabled=true
08/10/29 18:28:00 INFO common.Storage: Image file of size 96 saved in 0
seconds.
08/10/29 18:28:00 INFO common.Storage: Storage directory
/usr/local/hadoop-datastore/hadoop-hadoop/dfs/name has been successfully
formatted.
08/10/29 18:28:00 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at phil/127.0.1.1
************************************************************/

hadoop@phil:/usr/local/hadoop$ bin/hadoop dfs -put
/home/philippe/synthetic_control.data testdata

hadoop@phil:/usr/local/hadoop$ bin/hadoop jar
/home/philippe/workspace/MahoutJava/examples/build/apache-mahout-examples-0.1-dev.job
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
08/10/29 18:28:45 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/29 18:28:46 INFO mapred.FileInputFormat: Total input paths to process
: 1
08/10/29 18:28:47 INFO mapred.JobClient: Running job: job_200810291828_0002
08/10/29 18:28:48 INFO mapred.JobClient:  map 0% reduce 0%
08/10/29 18:28:54 INFO mapred.JobClient:  map 50% reduce 0%
08/10/29 18:28:55 INFO mapred.JobClient:  map 100% reduce 0%
08/10/29 18:28:56 INFO mapred.JobClient: Job complete: job_200810291828_0002
08/10/29 18:28:56 INFO mapred.JobClient: Counters: 7
08/10/29 18:28:56 INFO mapred.JobClient:   File Systems
08/10/29 18:28:56 INFO mapred.JobClient:     HDFS bytes read=291644
08/10/29 18:28:56 INFO mapred.JobClient:     HDFS bytes written=323660
08/10/29 18:28:56 INFO mapred.JobClient:   Job Counters
08/10/29 18:28:56 INFO mapred.JobClient:     Launched map tasks=2
08/10/29 18:28:56 INFO mapred.JobClient:     Data-local map tasks=2
08/10/29 18:28:56 INFO mapred.JobClient:   Map-Reduce Framework
08/10/29 18:28:56 INFO mapred.JobClient:     Map input records=600
08/10/29 18:28:56 INFO mapred.JobClient:     Map input bytes=288374
08/10/29 18:28:56 INFO mapred.JobClient:     Map output records=600
08/10/29 18:28:56 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/29 18:28:56 INFO mapred.FileInputFormat: Total input paths to process
: 2
08/10/29 18:28:56 INFO mapred.JobClient: Running job: job_200810291828_0003
08/10/29 18:28:57 INFO mapred.JobClient:  map 0% reduce 0%
08/10/29 18:29:03 INFO mapred.JobClient:  map 50% reduce 0%
08/10/29 18:29:05 INFO mapred.JobClient:  map 100% reduce 0%
08/10/29 18:29:10 INFO mapred.JobClient:  map 100% reduce 100%
08/10/29 18:29:11 INFO mapred.JobClient: Job complete: job_200810291828_0003
08/10/29 18:29:11 INFO mapred.JobClient: Counters: 16
08/10/29 18:29:11 INFO mapred.JobClient:   File Systems
08/10/29 18:29:11 INFO mapred.JobClient:     HDFS bytes read=323660
08/10/29 18:29:11 INFO mapred.JobClient:     HDFS bytes written=9657
08/10/29 18:29:11 INFO mapred.JobClient:     Local bytes read=36119
08/10/29 18:29:11 INFO mapred.JobClient:     Local bytes written=72300
08/10/29 18:29:11 INFO mapred.JobClient:   Job Counters
08/10/29 18:29:11 INFO mapred.JobClient:     Launched reduce tasks=1
08/10/29 18:29:11 INFO mapred.JobClient:     Launched map tasks=2
08/10/29 18:29:11 INFO mapred.JobClient:     Data-local map tasks=2
08/10/29 18:29:11 INFO mapred.JobClient:   Map-Reduce Framework
08/10/29 18:29:11 INFO mapred.JobClient:     Reduce input groups=1
08/10/29 18:29:11 INFO mapred.JobClient:     Combine output records=28
08/10/29 18:29:11 INFO mapred.JobClient:     Map input records=600
08/10/29 18:29:11 INFO mapred.JobClient:     Reduce output records=7
08/10/29 18:29:11 INFO mapred.JobClient:     Map output bytes=943020
08/10/29 18:29:11 INFO mapred.JobClient:     Map input bytes=323660
08/10/29 18:29:11 INFO mapred.JobClient:     Combine input records=1732
08/10/29 18:29:11 INFO mapred.JobClient:     Map output records=1732
08/10/29 18:29:11 INFO mapred.JobClient:     Reduce input records=28
08/10/29 18:29:11 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
08/10/29 18:29:11 INFO mapred.FileInputFormat: Total input paths to process
08/10/29 18:29:12 INFO mapred.JobClient: Running job: job_200810291828_0004
08/10/29 18:29:13 INFO mapred.JobClient:  map 0% reduce 0%
08/10/29 18:29:20 INFO mapred.JobClient:  map 50% reduce 0%
08/10/29 18:29:22 INFO mapred.JobClient:  map 100% reduce 0%
08/10/29 18:29:27 INFO mapred.JobClient:  map 100% reduce 100%
08/10/29 18:29:28 INFO mapred.JobClient: Job complete: job_200810291828_0004
08/10/29 18:29:28 INFO mapred.JobClient: Counters: 16
08/10/29 18:29:28 INFO mapred.JobClient:   File Systems
08/10/29 18:29:28 INFO mapred.JobClient:     HDFS bytes read=342974
08/10/29 18:29:28 INFO mapred.Jo

Hmm, I believe that patch has been applied in 18.2 (whatever that is)  
but it also looks like it has been applied to 0.17.3 branch as  
well.    So, it might be something else that "fixed" it.

At any rate, glad to hear it works on trunk.
On Oct 29, 2008, at 6:38 PM, Philippe Lamarche wrote:

Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Hi there,
It also works on 0.19.0-dev, that is on hadoop/branches/branch-0.19.

I intend in the next few day to try to find out what exactly is the problem
to make sure that it won't come back in a few revisions.

Thanks!

On Thu, Oct 30, 2008 at 9:20 AM, Grant Ingersoll <[EMAIL PROTECTED]>wrote:

Hi folks

A while back we upgraded our Hadoop cluster from 0.15 to 0.18.0, and I
found that Mahout Kmeans quit working.  I finally tracked it down to
the fact that the semantics of the combiner changed between 0.16,
0.17, and 0.18 from run exactly once to run zero or more times (which
is in line with how Map/Reduce was originally specified).  See:
https://issues.apache.org/jira/browse/HADOOP-3586.

The Kmeans combiner depended on running exactly once, but on our new
cluster it was running multiple times, causing hard-to-discern errors.
 Basically, the second time through the Combiner, it would throw an
exception that the formatting of the vector (serialized into a Text)
was failing.  In the end, I had to make some formatting changes to the
data output by the Mapper and the Combiner to match what the Reducer
expects, as well as changes to the Combiner input to .  I ended up
having to hack the Mapper to output vectors that either the Combiner
or Reducer could take as input, and make the Combiner take in the same
input that it outputs and to calculate convergence at each step.

My apologies if this has already been covered and put to rest - I just
happened upon this thread this afternoon.

-steve

On Sun, Nov 2, 2008 at 10:29 AM, Philippe Lamarche
<[EMAIL PROTECTED]> wrote:

Thanks Steve,

That was a subtle change that was evidently made after Kmeans was
implemented and did not show up until later when people such as Philippe
and yourself ran it with real problems on real clusters. While the type
signatures of the reducer and combiner are in fact the same, the values
provided by the mapper and combiner are different and could indeed
create the odd behavior that was reported.

The algorithm's dependence upon run-once behavior is pretty fundamental,
since summing of cluster centroids is done in the combiner and the
reducer does a merge of those clusters. I'd be interested in exactly how
you resolved this.

It likely applies to some of the other clustering implementations too.

Finally, can you explain why this problem no longer seems to occur with
Hadoop trunk?

Jeff
Steve Schlosser wrote:

Hi all,

 The same is discussed here:
https://issues.apache.org/jira/browse/MAHOUT-79

I have patch for fixing this issue ready. If no one is working on it, I
can open an issue in jira and commit the same.

Thanks
Pallavi

-----Original Message-----
From: Jeff Eastman [mailto:[EMAIL PROTECTED]]
Sent: Thursday, November 06, 2008 5:50 AM
To: [EMAIL PROTECTED]
Subject: Re: Problems with KMeans clustering

Thanks Steve,

That was a subtle change that was evidently made after Kmeans was
implemented and did not show up until later when people such as Philippe

and yourself ran it with real problems on real clusters. While the type
signatures of the reducer and combiner are in fact the same, the values
provided by the mapper and combiner are different and could indeed
create the odd behavior that was reported.

The algorithm's dependence upon run-once behavior is pretty fundamental,

since summing of cluster centroids is done in the combiner and the
reducer does a merge of those clusters. I'd be interested in exactly how

you resolved this.

It likely applies to some of the other clustering implementations too.

Finally, can you explain why this problem no longer seems to occur with
Hadoop trunk?

Jeff
Steve Schlosser wrote:problem<[EMAIL PROTECTED]>wrote:is) but
>>> it also looks like it has been applied to 0.17.3 branch as well.
So, it
>>> might be something else that "fixed" it.
>>>
>>> At any rate, glad to hear it works on trunk.
>>>
>>>
>>> On Oct 29, 2008, at 6:38 PM, Philippe Lamarche wrote:
>>>
>>>  I am not sure I understand the hadoop svn structure, however I was
able to
>>>      
>>>> make it work with hadoop trunk, or 0.20.0-dev.
>>>> It didn't work with hadoop/branch-0.18, with or without patch 4277.
>>>>
>>>>
>>>> Here is a copy-paste of the steps, once Hadoop is built and
installed.  I
>>>> am
>>>> using the same exact "apache-mahout-examples-0.1-dev.job", not
rebuilt
>>>> with
>>>> the 0.20.0-dev jars.
>>>>
>>>> It works!
>>>>
>>>> That would mean that the bug/feature is not related to
>>>> HADOOP-4277<http://issues.apache.org/jira/browse/HADOOP-4277>,
>>>>
>>>> and was reintroduced (or never took away) in hadoop/trunk.
>>>>
>>>>
>>>> hadoop@phil:/usr/local/hadoop$ bin/hadoop namenode -format
18:25:08
isPermissionEnabled=true
in 0
successfully
/home/philippe/workspace/MahoutJava/examples/build/apache-mahout-example
s-0.1-dev.job
for
same.
written=323660
for
same.
written=9657
written=72300
tasks=1
records=28
records=7
bytes=943020
records=1732
records=1732
records=28
for
same.
written=3002539
read=3018455
written=6036972
tasks=1
records=0
records=1591
bytes=3008903
records=0
records=1591
records=1591
for
same.
written=8205
written=46516
tasks=1
records=10
records=7
bytes=1136504
records=600
records=10
for
same.
written=8242
written=42592
tasks=1
records=10
records=7
bytes=1023966
records=600
records=10
for
same.
written=8280
written=42232
tasks=1
records=10
records=7
bytes=1023681
records=600
records=10
for
same.
written=8250
written=42740
tasks=1
records=10
records=7
bytes=1028419
records=600
records=10
for
same.
written=8200
written=42500
tasks=1
records=10
records=7
bytes=1024899
records=600
records=10
for
same.
written=1020535
for
same.
read=1020535
written=325460
bytes=1020535
week, but
18.2
have a
<[EMAIL PROTECTED]
either.
synthetic
just
any
and not
relying on
was
0.17.2 or
17.2?
of
traced
org.apache.hadoop.mapred.Merger:
org.apache.hadoop.mapred.Merger:
5011
inmemory
failed
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
MemMerge(ReduceTask.java:2147)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(
ReduceTask.java:2078)
string:
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1224)
org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
org.apache.mahout.matrix.AbstractVector.decodeVector(AbstractVector.java
org.apache.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner
.java:38)
org.apache.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner
.java:31)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier.combineAndSpill(ReduceT
ask.java:2174)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access$3100(ReduceTask.
java:341)
org.apache.hadoop.mapr

I think that would be good.  I'm going to be working on MAHOUT today  
and tomorrow, hopefully.  Finally have some free time at ApacheCon...

On Nov 5, 2008, at 11:08 PM, Palleti, Pallavi wrote: