Thanks for your reply Jagadish. We will certainly do some testing, I was just curious if anyone had tried or knew what we could expect. The broker format change is actually over 2 years old now, so I assumed someone had tried this by now ;)

-Tommy

On Mon, 2018-07-09 at 12:22 -0700, Jagadish Venkatraman wrote:

Hi Thomas,
Has Samza been tested against newer broker

versions using the new message format, and if so does it have a

significant performance impact?
We have not benchmarked Kafka broker performance with the new message

format.

Any benchmarking may not be reliably reproducible since there are many

variables (message sizes,

compressibility, how "saturated" the brokers are at the instant you run the

benchmark).
I'd suggest some general pointers on this.
   - First define the metric you're trying to optimize - broker-side

   throughput, Samza's throughput, broker-cpu utilization?

   - Specify what the acceptable value of the metric is for your current

   setup.

   - Then, measure it for your workload. For all you know, the performance

   might be "good enough"
Are their plans to move Samza to the

new consumer?
We certainly have plans to move to the "new" consumer for reasons unrelated

to throughput on the client-side

(eg: SSL, long-term support from the Kafka community). However, these plans

have not gotten enough traction.
Please let me know if there are further questions.
-- Jagadish
On Mon, Jul 9, 2018 at 9:47 AM, Thomas Becker <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>

wrote:
Anyone have any input here?
On Mon, 2018-07-02 at 11:50 +0000, Thomas Becker wrote:
Hey folks,
I have a question regarding potential performance impacts of running
Samza against newer Kafka brokers. We have languished on the old on-
disk message  format for Kafka for some time, and want to upgrade to
the newer format which supports timestamps. Samza currently accounts
for quite a bit of our message consumption and I am concerned that it
will cause a broker performance hit due to downconversion of messages.
I know Samza uses the old SimpleConsumer internally which does not
support the newer format. Has Samza been tested against newer broker
versions using the new message format, and if so does it have a
significant performance impact? Are their plans to move Samza to the
new consumer?

Regards,
Tommy Becker

________________________________

This email and any attachments may contain confidential and privileged

material for the sole use of the intended recipient. Any review, copying,

or distribution of this email (or any attachments) by others is prohibited.

If you are not the intended recipient, please contact the sender

immediately and permanently delete this email and any attachments. No

employee or agent of TiVo Inc. is authorized to conclude any binding

agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo

Inc. may only be made by a signed written agreement.

________________________________
This email and any attachments may contain confidential and privileged

material for the sole use of the intended recipient. Any review, copying,

or distribution of this email (or any attachments) by others is prohibited.

If you are not the intended recipient, please contact the sender

immediately and permanently delete this email and any attachments. No

employee or agent of TiVo Inc. is authorized to conclude any binding

agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo

Inc. may only be made by a signed written agreement.
________________________________

This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB