Kafka, mail # user - Re: S3 Consumer - 2012-12-28, 15:22
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hi Matthew,

I may be doing something wrong.

I cloned the code at
https://github.com/apache/kafka/tree/trunk/contrib/hadoop-consumer

I am running following :
- ./run-class.sh kafka.etl.impl.DataGenerator test/test.properties which
generates a /tmp/kafka/data/1.dat file containing
Dump tcp://localhost:9092       atlas-topic1    0       -1 to
/tmp/kafka/data/1.dat
- ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties
It says
Using offset range [0, 42649]
Connected to node tcp://localhost:9092 beginning reading at offset 0 latest
offset=42649
Again I run again
- ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties
It says
Using offset range [0, 42759]
Connected to node tcp://localhost:9092 beginning reading at offset 0 latest
offset=42759

My test.properties contain local file system for input/output to test

kafka.etl.topic=topic1
hdfs.default.classpath.dir=/tmp/kafka/lib
event.count=1000
hadoop.job.ugi=kafka,hadoop
kafka.server.uri=tcp://localhost:9092
input=/tmp/kafka/data
output=/tmp/kafka/output
kafka.request.limit=-1
client.buffer.size=1048576
client.so.timeout=60000

Pratyush
On Fri, Dec 28, 2012 at 8:02 PM, Matthew Rathbone <[EMAIL PROTECTED]>wrote:
Pratyush Chandra

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB