The Apache Pulsar community releases version 2.11! 61 contributors provided feature enhancements and fixes that delivered 1617 commits. Thanks for all your contributions.
Highlights of the 2.11 release
- Upgrade Pulsar server Java build version to 17 (PIP-156)
- Support chunking with shared subscriptions (PR-16202)
- Support pluggable topic factory (PR-12235)
- Support configurable compression type for
ManagedCursorInfoto balance storage resource usage (PR-14542)
This blog documents the most noteworthy changes in this release. For the complete list, including all feature enhancements and bug fixes, check out the Pulsar 2.11 Release Notes.
Notable feature enhancements and fixes
Broker
Store subscription properties within metadata service (PR-15757)
Issue
Restarting the broker or unloading topics may lead to the loss of subscription properties since they are not stored in the metadata service.
Resolution
Store the subscription properties within the metadata service (ManagedCursorInfo).
Support configurable compression type for ManagedCursorInfo (PR-14542)
Issue
When the cursor data expands, the data size increases and it takes a lot of time to pull the data.
Resolution
Introduce various compression algorithms to support cursor compression.
Reference documentation: managedCursorInfoCompressionType
Sync Pulsar metadata across multiple clouds (PR-16425)
Issue
Synchronizing configuration metadata (policies) is a critical path to sharing tenant/namespace/topic policies among clusters and administering policies uniformly across all clusters. However, syncing the metadata store (global zookeeper) among Pulsar clusters deployed on separate cloud platforms is not supported in earlier versions.
Resolution
Use two system topics metadataSyncEventTopic and configurationmetadataSyncEventTopic along with a metadata synchronizer to sync metadata among clusters deployed on different cloud platforms and persist local topic policies with it.
Reference documentation
Producer
Fix client memory limit currentUsage leak in ProducerImpl (PR-16837)
Issue
The client memory limit currentUsage leaks in ProducerImpl and causes the producer’s send rate to be slow.
Resolution
Release the memory when there are failed pending batch messages.
Release semaphore before discarding messages in batchMessageContainer (PR-17019)
Issue
There is a race condition in batchMessageContainer when discarding the messages and releasing the semaphore, which causes the semaphore to be held.
Resolution
Release the semaphore before discarding the messages in batchMessageContainer.
Consumer
Support consumer client memory limit (PR-15216)
Issue
It takes time for an application with a large number of producers/consumers to select an appropriate value for its queue size. The same applies to topics with many partitions.
Resolution
- Block the expansion of the consumer receiver’s queue size if the memory usage exceeds 75%.
- Trigger the shrinking of the consumer receiver’s queue size if the memory usage exceeds 95%.
Support adding interceptors for readers (PR-14729)
Issue
Pulsar supports adding interceptors for producers and consumers to implement message tracing, but there is no way to add interceptors for readers.
Resolution
- Add a new interface
ReaderInterceptorto customize the reader interceptor. - Support setting reader interceptor and auto-update partition configurations by using
ReaderBuilder.
User documentation: Create a reader with an interceptor
Function
Support Record as a new output type of Functions (PR-16041)
Issue
In earlier versions, if you want to dynamically set an output topic, message properties, or change the output schema in a Pulsar function, the only way is to create a function that returns Void. It would be more intuitive to return a structure like Record that carries this information.
Resolution
Add a utility method newOutputRecordBuilder to Context that returns a FunctionRecord builder initialized with the information from the source record. The builder methods can be used to override these values as needed.
User documentation: Use Record as function output
Tiered storage
Add a universal S3 provider for the offloader (PR-15710)
Issue
Pulsar supports cloud storage compatible with S3 APIs, such as AWS and Aliyun, through existing offloaders, but the forced registration of specific metadata limits their use.
Resolution
Provide a more general offloader S3 to serve more S3-compatible storage, which uses pure JClouds S3 metadata and allows overriding the default JClouds properties through system properties.
User documentation: Use S3 offloader with Pulsar
Proxy
Support PrometheusRawMetricsProvider for Pulsar proxies (PR-14681)
Issue
The metrics of plugins could not be exposed to Prometheus for monitoring.
Resolution
Add PrometheusRawMetricsProvider in Pulsar proxies so that plugins can add their metrics to Prometheus.
What’s Next?
If you are interested in learning more about Pulsar 2.11, you can download and try it out now!
For more information about the Apache Pulsar project and current progress, visit the Pulsar website, follow the project on Twitter @apache_pulsar, and join Pulsar Slack!