I'd seen references to there being a Kafka protocol dissector built into wireshark/tshark 1.12, but what I could find on that was a bit light on the specifics as to how to get it to do anything -- at least for someone (like me) who might use tcpdump a lot but who doesn't use tshark a lot.
I got this working, so I figured I'd post a few pointers here on the off-chance that they save someone else a bit of time.
Note that I'm using tshark, not wireshark; this might be easier and/or different in wireshark, but I don't feel like moving many gigabytes of data to a place where I can use wireshark. (-:
If you're reading traffic live, you'll want to do something like this:
You can get a complete list of Kafka-related fields by doing:
tshark -G fields | grep -i kafka
There is a very significant downside to processing packets live: tshark uses dumpcap to generate the actual packets, and unless I'm missing some obscure tshark option (which is possible!) it won't toss old data. So if you run this for a few hours, you'll end up with a ginormous file.
By default (under Linux, at least) tshark is going to put that file in /tmp, so if your /tmp is small and/or a tmpfs that can make things a little exciting. You can get around that by doing:
(export TMPDIR=/big/damn/filesystem ; tshark bla bla bla)
which I figure given typical Kafka data volumes is probably pretty important to know, and which doesn't seem to be documented in the tshark man pages. It is at least not all that hard to search for.
In theory, you can use the tshark "-b" option to specify a ring buffer of files, even for real-time processing, though:
* adding -b anything (e.g., "-b files:1 -b filesize:1024") seems to want to force you to use -w (filename)
* just adding -b and -w to the invocation above gets a warning about display filters not being supported when capturing and saving packets
* changing -Y to -2 -R and/or adding -P doesn't seem to help
(though again someone with more tshark experience might know the magic combination of arguments to get this to do what it's told).
So instead, you can capture packets somewhere, e.g.:
Thanks for sharing this, Steve! On Tue, Aug 12, 2014 at 11:03 AM, Steve Miller <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext