Event Hubs Consumer Groups and Checkpoints Storage

There are numerous questions/answers here on what the Consumer Group on the Event Hubs is. It looks like the short answer is:

Each posted event is delivered to one of the consumer clients in each consumer group. In other words, somebody in each group will see each message at least once. Some events might be/will be delivered multiple times.

I will assume that this explanation above as correct.

Now we have Checkpoints Storage as a separate entity. Hm. Why? What is missing in the Consumer groups? Why is this storage needed and why it is not inside the Event Hub Azure resource?

Next. Should all workers/clients in a Consumer group share the same checkpoints storage? What will happen if they will have their own? What will go wrong?

My understanding is that if workers/clients from different Consumer groups will share the same checkpoints storage, everything will be simply screwed up. Is that right?

Reading the docs my impression is that the checkpoints store controls who will see what. Then why the consumer groups are needed? Why the checkpoints stores are not enough? Why do we need both?

Side question. Does anybody know how can I have checkpoints storage not in the storage account? Is that possible with the standard .Net/C# Azure library? I want to create my own storage. There is even an abstract base class (Azure.Messaging.EventHubs.Primitives.CheckpointStore) for that.

Consumer groups helps event hubs to enable the publish/subscribe mechanism and provides a view for an event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets.

Each posted event is delivered to one of the consumer clients in each
consumer group. In other words, somebody in each group will see each
message at least once. Some events might be/will be delivered multiple
times.

This all true except for the last line. The events will be read only once. This is what the checkpoint ensures. Each message in the eventhub is available to every consumer group and will be read once (by default) by atleast 1 person/server within that consumer group. The event maybe read twice only if the person/consumer within the group specifically points to a different/earlier checkpoint.
So basically checkpoints are managed per partition and each consumer group should have their own checkpoint location.

Now we have Checkpoints Storage as a separate entity. Hm. Why? What is
missing in the Consumer groups? Why is this storage needed and why it
is not inside the Event Hub Azure resource?

Consumer groups need to keep track of what has been read per partition. This is why checkpoint storage is needed. It is provided as a config so that we can choose where we want the checkpoints to go rather than have it managed by Eventhub/Azure. Think of a scenario where we have a databricks on AWS reading from an Azure Eventhub and I want the checkpoints to be in my AWS bucket.

Next. Should all workers/clients in a Consumer group share the same
checkpoints storage? What will happen if they will have their own?
What will go wrong?

1 consumer group should have its own independent checkpoint storage. If you assign multiple consumers in the same consumer group separate checkpoints, multiple consumers will try to connect to a single partition. And this is essentially not good as they will be stepping on each others toes. Only 1 consumer in a consumer group can conntect to 1 partition at a time.
Each you have multiple consumers who need to read the same data and maybe at different speeds, put them on separate consumer groups and keep the checkpoint locations separate as well.

My understanding is that if workers/clients from different Consumer
groups will share the same checkpoints storage, everything will be
simply screwed up. Is that right?

Yes. Separate consumer groups are meant to read their own set of the same data. If all consumer groups share the same checkpoint storage, there is no parallelism other than partitions and is as good as 1 big consumer group having only 1 set/view of the data

Reading the docs my impression is that the checkpoints store controls
who will see what. Then why the consumer groups are needed? Why the
checkpoints stores are not enough? Why do we need both?

Consumer groups are needed incase of you have multiple consumers who need to see the same data. A typical example is a kafka pub sub model where you have multiple subscribers. Some maybe you have 1 databricks job doing read time aggregates but you also have some azure function triggered by your eventhub doing somehing for every single event. These 2 will be on separate consumer groups.

Side question. Does anybody know how can I have checkpoints storage
not in the storage account? Is that possible with the standard .Net/C#
Azure library? I want to create my own storage. There is even an
abstract base class
(Azure.Messaging.EventHubs.Primitives.CheckpointStore) for that.

No idea about this and not sure why you would want this when you have a managed storage provided by Azure.

Filed under: Kiến thức lập trình - @ 00:25

Thẻ: azureazure-eventhubazure-eventhub-client

Thiết kế website giá rẻ

Danh mục

Event Hubs Consumer Groups and Checkpoints Storage