Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Publishing/marking multiple event messages as single transaction?


Copy link to this message
-
Publishing/marking multiple event messages as single transaction?
Vjeran Marcinko 2014-02-12, 06:36
Hi,

 

And again, maybe this is more archtectural question, but I would like to
hear how people tackle this problem.

 

Anyway, as far as I can tell, a producer can publish multiple events as part
of one transaction on its side. Examples are:

-       Multiple entities are created as part of one action and we want to
report each entity's creation as one event message - CreateAccountCommand
being processed transactionally can result in creation of Account entitiy as
well as some Bonus entitiy that you get for free when you open your first
account (whatever that means); which means a producer publishes
AccountCreatedEvent and BonusCreatedEvent as part of that transaction

-       Sometimes an action being processed can publish not just event that
designates the action, but also the result of that processing - eg. In some
game engine, a player can do MovePlayerUpCommand (action), which publishes
PlayerMovedUp and only sometimes GameWonEvent if the move was crucial to
winning the game.

 

Thus if some consumer want to consume these events for sake of replicating
producer state, it should also be able to consume these events in batches
that correspond to original transaction, right? Otherwise, the consumer
would reconstruct state in some point in time that has never occurred in
producing side, which we surely don't want, right?

 

Anyone experienced this problem?

 

Or LinkedIn folks don't use Kafka for state replication, but Databus (which
maybe takes care of transaction thing - dunno, didn't researched it)?

 

Regards,

Vjeran