Types
Pulsar provides three types of broker load balancers:
Simple broker load balancer: it existed since Pulsar's inception, but now is deprecated.
The modular and extensible broker load balancer implements similar load balance functionalities with different system designs. For example, they both employ a similar approach to distributing data loads among brokers, including:
Dynamic bundle-broker assignment
Dynamic bundle splitting
Dynamic bundle unloading (shedding)
However, for bundle ownership and load data stores, the modular load balancer uses ZooKeeper, whereas the extensible load balancer uses System topics and Table views.
Side-by-side comparisons
| Dimension | Modular broker load balancer | Extensible broker load balancer |
|---|---|---|
| Available version | Pulsar 1.7.0 2017 | Pulsar 3.0.0 2023 |
| Load Balance Metadata Store and Replication | Dependent on ZooKeeper for load balance metadata store and replication. - All broker load data and all bundle load data are stored in ZooKeeper and replicated to all brokers via ZooKeeper watchers. - All bundle ownerships are stored in ZooKeeper as ephemeral locks. | Dependent on system topics and table views for load balance metadata store and replication. - All broker load data are stored in a non-persistent system topic and replicated to all brokers via table views. - Each broker publishes only top k bundles' load data in a non-persistent system topic, and only the leader broker consumes it via a table view. - Bundle ownerships are stored in a persistent system topic and replicated to all brokers via table views. Note: The absolute size of the replicated load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics. |
| Load Balance Decision (Load Balance leader dependency) | Single leader broker decides on bundle-broker assignment, bundle splitting, and bundle unloading. Note: The leader broker requires global load information | Each broker decides and runs bundle-broker assignment (lookup) and bundle splitting based on the local (replicated) information. The leader broker still decides on bundle unloading globally. |
| Pub/sub reconnection upon bundle unloading | When bundles are unloaded, the topics in the bundles are temporarily closed, and producers and consumers connect to new owner brokers. - Connecting to the new owner brokers involves redirecting lookup requests via the leader broker. | Any broker can respond to the lookup requests to locate the new owner brokers without asking the leader broker. The reconnection does not go through the leader broker. |
| Observability | You can monitor and analyze load balance with various metrics | It adds additional metrics and debug mode. For example: - Check the status of load balance decision breakdown and failure. - Use the dynamic debug mode configuration to dynamically turn on and off more load balance decision logs. |
Related topics
To get a comprehensive understanding and discover the key insights, see Broker load balancing | Overview.
To discover different usage scenarios, see Broker load balancing | Use cases.
To explore functionalities, see Broker load balancing | Features.
To understand advantages, see Broker load balancing | Benefits.
To learn essential fundamentals, see Broker load balancing | Concepts.
To get up quickly, see Broker load balancing | Quick start.
To migrate one broker load balancer type to another, see Broker load balancing | Migration.