Hi Thomas,

>> Has Samza been tested against newer broker
versions using the new message format, and if so does it have a
significant performance impact?

We have not benchmarked Kafka broker performance with the new message
format.
Any benchmarking may not be reliably reproducible since there are many
variables (message sizes,
compressibility, how "saturated" the brokers are at the instant you run the
benchmark).

I'd suggest some general pointers on this.

   - First define the metric you're trying to optimize - broker-side
   throughput, Samza's throughput, broker-cpu utilization?
   - Specify what the acceptable value of the metric is for your current
   setup.
   - Then, measure it for your workload. For all you know, the performance
   might be "good enough"

>> Are their plans to move Samza to the
new consumer?

We certainly have plans to move to the "new" consumer for reasons unrelated
to throughput on the client-side
(eg: SSL, long-term support from the Kafka community). However, these plans
have not gotten enough traction.

Please let me know if there are further questions.

-- Jagadish

On Mon, Jul 9, 2018 at 9:47 AM, Thomas Becker <[EMAIL PROTECTED]>
wrote:

> Anyone have any input here?
>
> On Mon, 2018-07-02 at 11:50 +0000, Thomas Becker wrote:
>
> Hey folks,
>
> I have a question regarding potential performance impacts of running
>
> Samza against newer Kafka brokers. We have languished on the old on-
>
> disk message  format for Kafka for some time, and want to upgrade to
>
> the newer format which supports timestamps. Samza currently accounts
>
> for quite a bit of our message consumption and I am concerned that it
>
> will cause a broker performance hit due to downconversion of messages.
>
> I know Samza uses the old SimpleConsumer internally which does not
>
> support the newer format. Has Samza been tested against newer broker
>
> versions using the new message format, and if so does it have a
>
> significant performance impact? Are their plans to move Samza to the
>
> new consumer?
>
>
> Regards,
>
> Tommy Becker
>
>
> ________________________________
>
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>

--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB