Skip to main content

Apache Pulsar 2.11.0

2023-01-11​

Important notice​

Starting from 2.11.0, the minimal version of JDK on the server side (broker, functions, connectors) is required to be 17. See Pulsar Runtime Java Version Recommendation and PIP-156 for more details.

  • Rename Pulsar txn metrics to specify OpenMetrics #16581
  • Rename Pulsar schema metrics to specify OpenMetrics #16610
  • Rename Pulsar lb metrics to specify OpenMetrics #16611
  • Remove jul-to-slf4j #16320
  • Fix get non-persistent topics issue in Namespaces #16170
  • Fix producer/consume permission can’t get schema #15956
  • Add verification for configured default backlog quota and retention #15441
  • Fix ineffective broker LoadBalance #15314
  • Ensure the deletion consistency of topic and schema #14608
  • Transaction cumulative ack redeliver change #14371
  • Bump prometheus client version from 0.5.0 to 0.15.0 #13785
  • [PIP-158] Split client TLS transport encryption from authentication #15634
  • [PIP-155] Remove Python 2 support #15376
  • [PIP-156] Build Pulsar Server on Java 17 #15264
  • [PIP-162] Enable system topic by default #15619

PIPs​

  • [PIP 37] Support chunking with Shared subscription #16202
  • [PIP-74] Pulsar client memory limits #14400
  • [PIP-81] Split createNewMetadataLedger into multiple methods for reuse #15425
  • [PIP 97] Update Authentication Interfaces to Include Async Authentication Methods #12104
  • [PIP-100] Support pluggable topic factory #12235
  • [PIP-105] Store Subscription properties #15757
  • [PIP-117] Change Pulsar standalone defaults #15478
  • [PIP-132] Include message header size when checking maxMessageSize for non-batch messages on the client side #14007
  • [PIP-136] Sync Pulsar metadata across multiple clouds #16425
  • [PIP-143] Support split bundle by specified boundaries #13796
  • [PIP-145] Improve performance of regex subscriptions #14505
  • [PIP-146] ManagedCursorInfo compression #14542
  • [PIP 149] Make the REST Admin API fully async #14365
  • [PIP-152] Support subscription level dispatch rate limiter setting #15295
  • [PIP-153] Optimize metadataPositions in MLPendingAckStore #15137
  • [PIP-160] Make transactions work more efficiently by aggregation operation for transaction log and pending ack store #15370
  • [PIP-161] Exclusive Producer: ability to fence out an existing Producer #15488
  • [PIP-163] Add lowWaterMark check before appending entry to TB #15424
  • [PIP-166] Add MANUAL delivery semantics for Functions #16279
  • [PIP-179] Support the admin API to check unknown request parameters #16135
  • [PIP-181] Pulsar Shell #16250
  • [PIP-184] Topic specific consumer priorityLevel #16715
  • [PIP-187] Add API to analyze a subscription backlog and provide an accurate value #16545
  • [PIP-189] No batching if only one message in batch #18548
  • [PIP-201] Extensions mechanism for Pulsar Admin CLI tools #17158

Broker​

  • [PIP 81] Split the individual acknowledgments into multiple entries

    • Add range set wrapper to support record dirty ledgers #15607
    • Notifications for faster topic discovery #16062
    • Split createNewMetadataLedger into multiple methods for reuse #15425

  • [PIP 105] Store Subscription properties

    • Store Subscription properties #15757
    • Pulsar Admin: create subscription with Properties #15503
    • Fix error on recycled SubscriptionPropertiesList #15335

  • [PIP 160] Make transactions work more efficiently by aggregation operation for transaction log and pending ack store

    • Pending ack log store enables the batch feature #16707
    • Metrics stats of Transaction buffered writer #16758
    • Transaction buffered writer supports Timer #16727
    • Transaction log store enables the batch feature #16685
    • Protocol changes and configuration changes for transaction batch log #16617
    • Txn buffered writer for transaction log batch #16428

  • [PIP 145] Improve performance of regex subscriptions

    • Enable evaluating subscription pattern on broker side #14804
    • Notifications for faster topic discovery #16062

  • [PIP-156] Build Pulsar Server on Java 17

    • Build changes by dropping some required checks #15496
    • Build Pulsar Server on Java 17 #15264

  • Add logs for why namespace bundle been split #19003

  • Omit making a copy of CommandAck when there are no broker interceptors #18997

  • Fix deadlock in PendingAckHandleImpl #18989

  • Copy proto command fields into final variables in ServerCnx #18987

  • Transaction pending ack store future not completely problem #18943

  • Update interceptor handler exception #18940

  • Do not require encryption on system topics #18898

  • Fix incorrect Nic usage collected by pulsar #18882

  • Fix PendingAckHandleImpl when pendingAckStoreProvider.checkInitializedBefore failed #18859

  • Ignore the exception of creating namespace #18837

  • Fixing can not delete namespace by force #18826

  • Avoid OOM not trigger PulsarByteBufAllocator outOfMemoryListener when use ByteBufAllocator.DEFAULT.heapBuffer #18747

  • System topic writer/reader connection not counted #18603

  • DnsResolverUtil.TTL should be greater than zero #18565

  • Persist correct markDeletePosition to prevent message loss #18237

  • Fix delete_when_subscriptions_caught_up doesn't work while have active consumers #18320

  • Read local cookie when start pulsar standalone #18260

  • Remove the redundant judgment logic of ManagedCursorImpl #18205

  • Fix unexpected behaviour by invoke PulsarConfigurationLoader#convertFrom #18816

  • Fix the wrong behaviour when set overrideBrokerNicSpeedGbps #18818

  • Fix the breaking change of standalone metadata initialization #18909

  • Transaction buffer recover blocked by readNext #18969

  • Fix duplicated schemas creation #18701

  • Read local cookie when start pulsar standalone #18260

  • The Pulsar standalone bookie is not getting passed the config from standalone.conf #18126

  • Reduce unnecessary calling span() when filtering read entries #18106

  • Change update schema auth from tenant to produce #18074

  • Fixed delayed delivery after read operation error #18098

  • Fix memory leak while Offloading ledgers #18500

  • Cache invalidation due to concurrent access #18076

  • Fix unable to start multiple bookies for BKCluster #18072

  • Compare batch index when accumulating acks and updating batchDeletedIndexes #18042

  • Limit the memory used by reads end-to-end #18245

  • Consumer backlog eviction policy should not reset read position for consumer #18037

  • Allow to configure and disable the size of lookahead for detecting fixed delays in messages #17907

  • Fix create ns #17864

  • Fix the wrong NIC speed rate unit #17890

  • Fix numerical overflow bug while reading data from tiered storage #18595

  • Support lower boundary shedding for ThresholdShedder #17456

  • Avoid unnecessary creation of BitSetRecyclable objects #17998

  • Fix broker cache eviction of entries read by active cursors #17273

  • Fix issue where leader broker information isn't available after 10 minutes #17401

  • Fix NPE when ResourceGroupService execute scheduled task #17840

  • Make BookieId work with PulsarRegistrationDriver #17762

  • Fix namespace backlog quota check with retention #17706

  • Cleanup state when lock revalidation gets LockBusyException #17700

  • Fix parsing partitionedKey with Base64 encode issue #17687

  • Standalone Add param of --metadata-url for runing with metadata #17077

  • Correctly set byte and message out totals per subscription #18451

  • Implementation of Delayed Transaction Messages #17548

  • Set revalidateAfterReconnection true for certain failures #17664

  • Unregister topic policy listener if managed ledger close failed #17652

  • Correctly handle list configuration values #17661

  • Cancel the loadShedding task when closing pulsar service #17632

  • Consumer can't consume messages because there has two sames topics in one broker #17526

  • Prevent partitioned metadata lookup request when broker is closing #17315

  • Prevent new connection request when broker is closing #17314

  • Ledger handle leak when update schema #17283

  • Add metrics for entry cache insertion, eviction #17248

  • Do not send duplicate reads to BK/offloaders #17241

  • Fix system service namespace create internal event topic #17867

  • Fix getPositionAfterN infinite loop #17971

  • Fix memory leak in case of error conditions in PendingReadsManager #17995

  • After the broker is restarted, the cache dynamic configuration is invalid #17035

  • Fix if dynamicConfig item in ZK do not exist in broker cause NPE #17705

  • Fix the broker shutdown issue after Zookeeper node crashed #17909

  • Fix broker irrational behavior when it is closing #17085

  • ServerCnx: log at warning level when topic not found #16225

  • Optimize getting ledger and entry id from entry #17108

  • Topic policy reader can't recover when get any exception #17562

  • Multiple consumer dispatcher stuck when unackedMessages greater than maxUnackedMessages #17483

  • Fix memoryLimitController currentUsage and MaxQueueSize semaphore leak when batchMessageContainer add message exception #17276

  • Fix arithmetic exception for LeastResourceUsageWithWeight strategy #17149

  • Fix update topic remove properties #17231

  • Fix ack with txn compute ackedCount error #17016

  • Improve cursor.getNumberOfEntries if isUnackedRangesOpenCacheSetEnabled=true #17465

  • Fix dispatch duplicated messages with Exclusive mode #17237

  • Remove internalUpdateOffloadPolicies to keep the same behavior update topic policy #17236

  • Fix update topic remove properties #17231

  • Fix arithmetic exception for LeastResourceUsageWithWeight strategy #17149

  • Fix pulsarLedgerIdGenerator can't delete index path when zk metadata store config rootPath #17192

  • Avoid messages being repeatedly replayed with SHARED subscriptions (streaming dispatcher) #17163

  • Add ServerCnx state check before server handle request #17084

  • Pass subscriptionName to auth service #17123

  • Support loadBalancerSheddingIntervalMinutes dynamic configuration #16408

  • Fix out of order data replication #17154

  • Fix schema does not replicate successfully #17049

  • Optimize the shutdown sequence of broker service when it close #16756

  • Modernizer managed ledger #16363

  • Make deleteTopicPolicies serialized is executed when close topic #15811

  • Streaming dispatcher stuck after reading the first entry with SHARED subscriptions #17143

  • Fix calculate avg message per entry #17046

  • Fix bundle-data metadata leak because bundle stats was not cleaned up #17095

  • Duplicate ByteBuffer when Caching Backlogged Consumers #17105

  • Fix offload read handle NPE #17056

  • Increment topic stats outbound message counters and update rate after messages have been written to the TCP/IP connection #17043

  • Move the state check forward #17020

  • Follow up on #16968 to restore some behavior in PersistentDispatcherMultipleConsumers #17018

  • Remove exception log when access status.html #17025

  • Not allow terminating system topic #17006

  • Fix memory leak if entry exists in cache #16996

  • Remove unnecessary lock on the stats thread #16983

  • Prevent StackOverFlowException in SHARED subscription #16968

  • Improve naming for delete topic error #16965

  • Fix ConcurrentModificationException when ModularLoadManagerImpl start #16953

  • Skip mis-configured resource usage(>100%) in load balancer #16937

  • Adapt basic authentication configuration with prefix #16935

  • Change delete pending ack position from foreach to firstKey #16927

  • Fix MaxQueueSize semaphore release leak in createOpSendMsg #16915

  • PulsarLedgerManager to pass correct error code to BK client #16857

  • Support start multiple bookies for BKCluster #16847

  • Split TLS transport encryption support from authentication #16819

  • Fix misleading -c option in pulsar standalone #16838

  • Fix rack awareness cache expiration data race #16825

  • Fix Repeated messages of shared dispatcher #16812

  • Avoid IllegalStateException while client_version is not set #16788

  • Optimize concurrent collection's shrink logic #16754

  • Fix consumer does not abide by the max unacks limitation for Key_Shared subscription #16718

  • Support clear old bookie data for BKCluster #16744

  • Avoid ConcurrentModificationException for ModularLoadManagerImpl.cleanupDeadBrokersData() #16690

  • Improve get the basic authentication config #16526

  • Retry to delete the namespace if new topics created during the namespace deletion #16676

  • Fix consumer does not abide by the max unacks limitation for Shared subscription #16670

  • Add decode InputStream for Schema #16659

  • Support for get token from HTTP params #16650

  • Make defaultOffloader create after offloaderStats overrides #16638

  • BadVersionException when splitting bundles, delay 100ms and try again #16612

  • The configuration loadBalancerNamespaceMaximumBundles is invalid #16552

  • PulsarLedgerManager: add missed return statement #16607

  • Retry when DistributedIdGenerator has BadVersion error #16491

  • Fixed error when delayed messages trackers state grows to >1.5GB #16490

  • Fix RawReader hasMessageAvailable returns true when no messages #16443

  • Fix No such ledger exception #16420

  • Improve the package download process #16365

  • Add config maxUnloadBundleNumPerShedding for UniformLoadShedder #16409

  • Skip reading more entries for a pending read with no more entries #16400

  • Recycle OpReadEntry in some corner cases #16399

  • Add dynamic configuration for UniformLoadShedder #16391

  • Fix RawReader out of order #16390

  • Create the cursor ledger lazily to improve the subscribe performance #16389

  • Release the entry in getEarliestMessagePublishTimeOfPos #16386

  • Update CPU ResourceUsage before updating SystemResourceUsage usage #16366

  • Use shared broker client scheduled executor provider #16338

  • Fix etcd cluster error and add test for etcd cluster #16309

  • Do not use IO thread for consumerFlow in Shared subscription #16304

  • Provide new load balance placement strategy implementation based on the least resource usage with weight #16281

  • Fix TopicTransactionBuffer ledger apend marker throw ManagedLedgerAlreadyClosedException #16265

  • Avoid go through all the consumers to get the message ack owner #16245

  • Reduce the consumers list sort by priority level #16243

  • Reduce the re-schedule message read operation for PersistentDispatcherMultipleConsumers #16241

  • Fix NPE when invoke replaceBookie. #16239

  • Fix getInternalStats occasional lack of LeaderInfo again #16238

  • Fix NPE when drop backlog for time limit #16235

  • Improve error msg when client is unauthorized #16224

  • Fix compaction subscription acknowledge Marker msg issue #16205

  • Fix subscribe dispatcher limiter not be initialized #16175

  • Using handle instead of handleAsync to avoid using common pool thread #17403

  • Use LinkedHashSet for config items of type Set to preserve elements order #16138

  • Fix topic dispatch rate limiter not init on broker-level #16084

  • Fix NPE when get /admin/v2/namespaces/public/default/maxTopicsPerNamespace #16076

  • Add config to allow deliverAt time to be strictly honored #16068

  • Add switch for enable/disable distribute bundles evenly in LoadManager #16059

  • Fix thread safety issues in accessing ManagedCursorContainer.heap ArrayList #16049

  • Make invalid namespace and topic name logs more descriptive #16047

  • Terminate JVM when initialize-cluster-metadata command fails #16039

  • Avoid storing MessageMetadata instances returned by peekMessageMetadata #15983

  • Fix topic policies update not check message expiry #15941

  • Fix reach max tenants error if the tenant already exists #15932

  • Close transactionBufferClient before closing the internal Pulsar client #15922

  • Remove topic -> namespace cache of LedgerOffloaderStatsImpl #15869

  • Fix NPE when ledger id not found in OpReadEntry #15837

  • Fix topic-level replicator rate limiter not init #15825

  • Fix NPE in MessageDeduplication #15820

  • Configure DLog Bookie, Pulsar, and Admin clients via pass through config #15818

  • Fix the broker close hanged issue #15755

  • Disable memory limit controller in internal Pulsar clients #15752

  • Add timeout for unload namespace bundle #15719

  • Support schemaValidationEnforced on topic level #15712

  • Fix NPE when put value to RangeCache #15707

  • Fast return if ack cumulative illegal #15695

  • Fix creating producer failure when set backlog quota #15663

  • Fix logging in shutdown when broker shutdown #15626

  • Fix MetadataStoreException$NotFoundException while doing topic lookup #15633

  • Use dispatchRateLimiterLock to update dispatchRateLimiter #15601

  • Add .yaml suffix support for broker filter config file #15600

  • Add newline to the end of standalone.conf #15591

  • Create "standalone" cluster when starting standalone server #15583

  • Close publishLimiter when disable it #15520

  • Enable SO_REUSEADDR on listen sockets for broker,proxy #15517

  • Fix to avoid TopicStatsImpl NPE even if producerName is null #15502

  • Close connection if a ping or pong message cannot be sent #15382

  • Introduce BitRateUnit for calculate bitrate #15435

  • Add verification for configured default backlog quota and retention #15441

  • Change unnecessary atomic to basic type #15369

  • Support shrink for ConcurrentSortedLongPairSet #15354

  • Use shrink map for message redelivery #15342

  • Fix precision error in UsageUnit #15322

  • Optimized namespace-level dispatcherRateLimiter update #15315

  • Fix wrong unit of NIC speed on Linux #15304

  • Fix totalEntries calculation problem in AbstractBaseDispatcher#filterEntriesForConsumere #15298

  • Fix resource group does not report usage #15292

  • Tidy up the system topic #15252

  • Fix race condition between timeout and completion in OpAddEntry #15233

  • Improve skipping of DNS resolution when creating AuthenticationDataHttp instance #15228

  • Skip unnecessary DNS resolution when creating AuthenticationDataHttp instance #15221

  • Fix MessageDeduplication#inactiveProducers may not be persistence correctly #15206

  • Cancel fencedTopicMonitoringTask when topic closed normally #15202

  • Fix npe when doCacheEviction #15184

  • Make health check fail if dead locked threads are detected #15155

  • Fix parameter saslJaasBrokerSectionName in broker.conf #15110

  • Fix potential to add duplicated consumer #15051

  • Fix rewind failed when redeliverUnacknowledgedMessages #15046

  • Fix race condition in updating lastMarkDeleteEntry field #15031

  • Avoid heartbeat topic to offload #15008

  • Evict cache data according to the slowest markDeletedPosition #14985

  • Return if reset in progress #14978

  • Dynamic update broker-level subscribe-rate limter #14890

  • Fix wrong state for non-durable cursor #14869

  • Support advertised listeners for HTTP and HTTPS services #14839

  • Support dynamic update between non-zero values of topicPublisherThrottlingTickTimeMillis #14782

  • Support dynamic update between non-zero values of brokerPublisherThrottlingTickTimeMillis #14759

  • Fix incorrect entryId in warn log when reading entry from tiered storage #14685

  • Optimize PositionImpl toString, compareTo and hashCode methods #14678

  • Improve readability of ManagedCursorImpl source code #14617

  • Intercept ACK command by BrokerInterceptor #14616

  • Add filteredMsgCount for pulsar-admin broker-stats topics #14531

  • Fix NPE when subscription is already removed #14363

  • Fix backpressure handling in Jetty web server configuration #14353

  • Optimize find nics process #14340

  • Support pass http auth status #14044

  • Support caching to drain backlog consumers #12258

  • Strict bookie affinity group strategy #12025

Clients​

  • [PIP 74] Pulsar client memory limits
    • Support auto scaled consumer receiver queue #14494
    • Support dynamic limit of consumer receiver queue #14400
    • Support consumer client memory limit #15216
  • [Java] Fix multi-topic consumer stuck after redeliver messages #18491
  • [Java] Fix NPE of MultiTopicsConsumerImpl due to race condition #18287
  • [Java] Refactor SchemaHash to reduce call of hashFunction in SchemaHash #17948
  • [Java] Fix pendingLookupRequestSemaphore leak when channel inactive #17856
  • [Java] Unwrap completion exception for Lookup Services #17717
  • [Java] Fixed ack failure in ReaderImpl due to null messageId #17728
  • [Java] Fix scheduledExecutorProvider not shutdown #17527
  • [Java] LastBatchSendNanoTime initialization #17058
  • [Java] Fix inactive cnx channel causing the request to fail to time out and fail to return #17051
  • [Java] Release semaphore before discarding messages in batchMessageContainer #17019
  • [Java] Fix client memory limit currentUsage leak and semaphore release duplicated in ProducerImpl #16837
  • [Java] Remove redundant check for chunked message TotalChunkMsgSize in ConsumerImpl #16797
  • [Java] Reduce code duplication in admin client #16377
  • [Java] Release memory usage for invalid messages #16835
  • [Java] Fix subscription topic name error #16719
  • [Java] Send CloseConsumer on client timeout #16616
  • [Java] Add message key if exists to deadLetter messages #16615
  • [Java] Make DeadLetterPolicy deserializable #16513
  • [Java] Improve performance of multi-topic consumer with more than one IO thread #16336
  • [Java] Add initialization for the OpSendMsg #16256
  • [Java] Replace ScheduledExecutor to improve performance of message consumption #16236
  • [Java] Fix large messages sometimes cannot be split into chunks #16196
  • [Java] Only trigger the batch receive timeout when having pending batch receives requests #16160
  • [Java] Fix the startMessageId can't be respected as the ChunkMessageID #16154
  • [Java] Fix auto cluster failover can't resolve host bug #16152
  • [Java] Switch to rely on Netty for Hostname Verification #15824
  • [Java] Prevent the trigger from running concurrently in the memory limit controller #15737
  • [Java] Fix messages sent by producers without schema cannot be decoded #15622
  • [Java] Remove sensitive msg from consumer/producer stats log #15483
  • [Java] Add pending messages information while printing the producer stats #15440
  • [Java] Fix flaky BatchMessageTest by initializing lastBatchSendNanoTime #15406
  • [Java] Check consumer schema null in advance #15327
  • [Java] Fix inconsistent parameter of TopicPolicies.getSubscriptionDispatchRate #15293
  • [Java] Fix typo in ConsumerBuilder #15194
  • [Java] Fix performance regression with message listener #15162
  • [Java] ConsumerBuilderImpl can not set null to deadLetterPolicy #14980
  • [Java] Fix message publishing stuck when enabling batch #14870
  • [Java] Return immutable data set when using TableView #14833
  • [Java] Avoid timer task run before previous subscribe complete #14818
  • [Java] Support Reader Interceptor #14729
  • [Java] Support aggregate metrics for partition topic stats #18214
  • [Java] Add api to get producer/consumer stats for partition topic #18212
  • [Java] Optimize pause when creating sub consumers in multi-topic consumer #14566

Pulsar IO and Pulsar Functions​

  • [Functions] Fix function failed to start if no typeClassName provided in FunctionDetails #18111
  • [Functions] Do not delete managed package when delete function #18030
  • [Functions] Fix the download of builtin Functions #17877
  • [Functions] Ensure InternalConfigurationData data model is compatible across different versions #17690
  • [Functions] Use the schema set by the Function when it returns a Record #17142
  • [Functions] Make mandatory to provide a schema in Context::newOutputRecordBuilder #17118
  • [Functions] Add the ability to customize logging level for Go & Python functions #16939
  • [Functions] Fixed error when user starts with the pulsar functions local runner #16565
  • [Functions] Fix netty.DnsResolverUtil compat issue on JDK9+ for the function Runtimes #16423
  • [Functions] Ensure bytes are a well-formed UTF-8 byte sequence when decoding the FunctionState bytes to string #16199
  • [Functions] Support Record<?> as Function output type #16041
  • [Functions] Configure DLog Bookie, Pulsar, and Admin clients via pass through config #15818
  • [Functions] Disable memory limit controller in internal Pulsar clients #15752
  • [Functions] Provide default error handler for function log appender #15728
  • [Functions] Support disabling non-TLS service port #15328
  • [Functions] Check executor null when close the FileSource #15247
  • [Functions] Fix UserConfigFunction example #15240
  • [Functions] Add a cache of versioned KeyValueSchemaImpl #15122
  • [Functions] Add KeyStore support in WebSocket, Function Worker HTTPS Servers #15084
  • [Functions] Handle NPE when getLeader returns null #15058
  • [Functions] Remove internal dependency: pulsar-functions-instance #14925
  • [Functions] Allow a Function<GenericObject,?> to access the original Schema of the Message and use it #14847
  • [Functions] Fix pulsar-managed runtimes failed start function with package URL from package management service #14814
  • [Functions] Set function channel to idle to prevent DNS resolution of deleted pod #14750
  • [Functions] Remove extra call to gi.stats.incrTotalProcessedSuccessfully() #12316
  • [Functions] Fix python instance not process zip file correctly #16697
  • [Functions] Missing assertion in KubernetesRuntimeTest::verifyJavaInstance #16747
  • [IO Connector] Only bundle kafka schema registry client #18931
  • [IO Connector] Fix message without schema issue #18745
  • [IO Connector] Kinesis sink: fix NPE with KeyValue schema and no value #17959
  • [IO Connector] Provide a HTTP Sink #17581
  • [IO Connector] IOConfigUtils support required and defaultValue annotations #16785
  • [IO Connector] Fixed Kafka source config for pulsar-io-kafka when consumerConfigProperties is null #16731
  • [IO Connector] Add reader config to pulsar-io-debezium and pulsar-io-kafka-connect-adaptor #16675
  • [IO Connector] Support transactions for JDBC connector #16468
  • [IO Connector] JDBC sinks: support upsert and row deletion #16448
  • [IO Connector] Add getSourceConfig method on SourceContext #16305
  • [IO Connector] KCA sinks: fix offset mapping when sanitizeTopicName=true #15950
  • [IO Connector] Allow pulsar_tool_env.sh PULSAR_MEM to be Overridden #15868
  • [IO Connector] JDBC sinks: support Avro specific datatypes #15845
  • [IO Connector] Sink support custom acknowledge type #15491
  • [IO Connector] Add getSinkConfig method to SinkContext #15482
  • [IO Connector] Handle Avro collections native types (GenericData.Array and Utf8 map keys) #15432
  • [IO Connector] Handle Avro collections native types (GenericData.Array and Utf8 map keys) #15430
  • [IO Connector] Add hashed id support #15428
  • [IO Connector] ElasticSearch Sink: option to output canonical key fields (JSON and Avro) #15426
  • [IO Connector] Replaced "Device" with "measurement" #15416
  • [IO Connector] Enable bulk flushing scheduling by default #15244
  • [IO Connector] Set sinkType in config file #15174
  • [IO Connector] Add a cache of versioned KeyValueSchemaImpl #15122
  • [IO Connector] Support 4paradigm/OpenMLDB jdbc sink connector #15064
  • [IO Connector] ElasticSearch Sink: option to disable SSL certificate validation #14997
  • [IO Connector] Add FULL_MESSAGE_IN_JSON_EXPAND_VALUE message format to Kinesis sink #14929

Observability​

  • [Broker] Fix metrics string encoding #18138
  • [Broker] Fix EntryFilter stats #17605
  • [Broker] Add per-subscription EntryFilter metrics #16932
  • [Broker] Expose topic level storage write and read rate metrics #16855
  • [Broker] Rename Pulsar lb metrics to specify OpenMetrics #16611
  • [Broker] Rename Pulsar schema metrics to specify OpenMetrics #16610
  • [Broker] Rename Pulsar txn metrics to specify OpenMetrics #16581
  • [Broker] Optimise msgOutCounter and bytesOutCounter #16214
  • [Broker] Fail to expose managed ledger client stats to prometheus if bookkeeperClientExposeStatsToPrometheus is true #16219
  • [Broker] Add metrics for pulsar web service thread pool #14742
  • [Broker] Add support of PrometheusRawMetricsProvider for the Pulsar-Proxy #14681
  • [Broker] Add filteredMsgCount for pulsar-admin broker-stats topics #14531
  • [Broker] Improve /metrics endpoint performance #14453
  • [Broker] Offloader metrics #13833
  • [Broker] Improve /metrics endpoint performance #14453
  • [Broker] Add metrics for InMemoryDelayedDeliveryTracker's memory usage #15867

CLI​

  • [Pulsar Admin] Fix NPE when get OffloadThreshold on namespace #18061
  • [Pulsar Admin] Add SNI header when tlsHostnameVerification is not enabled #17543
  • [Pulsar Admin] Unset namespace policy to improve deleting namespace #17033
  • [Pulsar Admin] Reduce PartitionedStats local REST call #16916
  • [Pulsar Admin] Make some police methods async in Namespaces #16881
  • [Pulsar Admin] Expose the last consumed flow timestamp for consumer stats #16817
  • [Pulsar Admin] Make some methods async in Namespaces #16814
  • [Pulsar Admin] Dynamic configuration for check unknown request parameters #16781
  • [Pulsar Admin] Make offload police methods async in Namespaces #16760
  • [Pulsar Admin] Support the admin API to check unknown request #16577
  • [Pulsar Admin] Make getBacklogQuotaMap method async in Namespaces #16504
  • [Pulsar Admin] Make GetMessageIdByTimestamp pure async #16446
  • [Pulsar Admin] Make splitNamespaceBundle and getTopicHashPositions async #16411
  • [Pulsar Admin] Make AutoSubscriptionCreation async #16329
  • [Pulsar Admin] Make SubscriptionExpirationTime method async #16328
  • [Pulsar Admin] Make some method on permission async #16324
  • [Pulsar Admin] Make unloadNamespaceBundle async #16313
  • [Pulsar Admin] Make Namespaces.deleteNamespaceBundle async #16287
  • [Pulsar Admin] Make deleteTopic method async #16232
  • [Pulsar Admin] Make compactionStatus method async #16231
  • [Pulsar Admin] Make terminate method async #16227
  • [Pulsar Admin] Make getList async #16221
  • [Pulsar Admin] Make getPartitionedTopicList method async #16217
  • [Pulsar Admin] Improve documentation for unackedMessages stat #16213
  • [Pulsar Admin] Make internalPeekNthMessage method async #16192
  • [Pulsar Admin] Fix get non-persistent topics issue in Namespaces #16170
  • [Pulsar Admin] Make getInternalStats method async #16141
  • [Pulsar Admin] Fix get-publish-rete Admin API handle exception behavior #16001
  • [Pulsar Admin] Release LookupRequestSemaphore before returning data #15974
  • [Pulsar Admin] Fix producer/consume permission can’t get schema #15956
  • [Pulsar Admin] Make publish rate and dispatch rate operation async #15946
  • [Pulsar Admin] Support to get topic properties #15944
  • [Pulsar Admin] New API to get subscription properties #16095
  • [Pulsar Admin] Make some operation subscription dispatchRate methods in Namespaces async #15880
  • [Pulsar Admin] Make some methods in SchemasResourceBase async #15821
  • [Pulsar Admin] Make some operation replication clusters async #15760
  • [Pulsar Admin] Make some methods of ClusterBase pure async #15685
  • [Pulsar Admin] Allow creating builtin functions in pulsar-admin CLI #15671
  • [Pulsar Admin] Enable users to specify TTL options in units other than seconds #15657
  • [Pulsar Admin] Make some operation SubscribeRate methods in Namespaces async #15656
  • [Pulsar Admin] Add transaction admin to get recover time in stats #15654
  • [Pulsar Admin] Make some operation auto topic creation in Namespaces async #15621
  • [Pulsar Admin] Add topic name and sub name for NotFound error message #15606
  • [Pulsar Admin] Make some methods in ResourceQuotasBase async #15605
  • [Pulsar Admin] Make some methods in TenantsBase async #15603
  • [Pulsar Admin] Make some operation messageTTL methods in Namespaces async #15577
  • [Pulsar Admin] Enable users to specify some time options in units other than seconds #15563
  • [Pulsar Admin] Make some methods of ClusterBase pure async #15527
  • [Pulsar Admin] Support filtering system topic when get list #15410
  • [Pulsar Admin] Make some methods of ClusterBase pure async #15358
  • [Pulsar Admin] Make some methods of ClusterBase pure async #15318
  • [Pulsar Admin] Add admin api updateTransactionCoordinatorNumber #15296
  • [Pulsar Admin] Put validateTopicOwnershipAsync before validateTopicOperationAsync #15265
  • [Pulsar Admin] Remove duplication validateTopicOwnershipAsync #15120
  • [Pulsar Admin] Fix pulsar-admin not prompting message when there is a 500 error #14856
  • [Pulsar Admin] Add get active brokers api without cluster name #14702
  • [Pulsar Admin] Grab contextual stacktrace for sync methods #14620
  • [Pulsar Admin] Correct status message prompt #14603
  • [Pulsar CLI] Fix NPE in admin-CLI topic stats command #18326
  • [Pulsar CLI] Fix CLI client produce don't able to use multiple -m send multiple messages #18238
  • [Pulsar CLI] Pulsar shell: allow to create a new config (--file) with a relative path #17675
  • [Pulsar CLI] Pulsar shell: do not exit on user interrupt during commands #18615
  • [Pulsar CLI] Pulsar shell: allow cloning an existing config #18608
  • [Pulsar CLI] Pulsar shell: support relative paths #17648
  • [Pulsar CLI] Pulsar shell, Pulsar admin, Pulsar client and Pulsar perf: support for Windows #17243
  • [Pulsar CLI] Pulsar shell: add command to set/get property of a config #17651
  • [Pulsar CLI] Fix Pulsar shell custom commands execution #17479
  • [Pulsar CLI] CLI extensions: rename default location to 'cliextensions' #17435
  • [Pulsar CLI] Exclude windows script from the docker image #17404
  • [Pulsar CLI] Add a separate TLS transport encryption configuration #16930
  • [Pulsar CLI] Fix incorrect description for producing messages #16876
  • [Pulsar CLI] Use NoSplitter for subscription properties #16862
  • [Pulsar CLI] Add TLS provider support #16700
  • [Pulsar CLI] Support getEarliestTimeInBacklog at getPartitionedStats method #16388
  • [Pulsar CLI] Add query options to the get topics in Namespace #16167
  • [Pulsar CLI] Add cli cmd for subscription level dispatch-rate-limiter #15862
  • [Pulsar CLI] Disable Pulsar client memory limit by default #15748
  • [Pulsar CLI] Fix check on create function class name #15700
  • [Pulsar CLI] Support filtering system topic when get list #15410
  • [Pulsar CLI] Change the default for max-connections from 100 to 1 #15387
  • [Pulsar CLI] Add short name for full name in admin cli #14301
  • [Client Tool] Using int instead of long in python scripts #17215
  • [Client Tool] Fix using directory '?' in the docker image #17185
  • [Client Tool] Handle ctrl-d and exit the shell #17204
  • [Client Tool] Add tlsTrustCertFilePath as CLI argument for pulsar-client-tool #16961
  • [Perf Tool] Able to set maxLookupRequest for pulsar-perf #16967
  • [Perf Tool] Pulsar-perf fails on jdk17 #18806

Others​

  • [Proxy] Fix refresh client auth #17831
  • [Proxy] Update proxy lookup throw exception type #17600
  • [Proxy] Add TLS transport encryption for broker client #16833
  • [Proxy] Fix client service URL #16834
  • [Proxy] Consolidate Netty channel flushes to mitigate syscall overhead #16372
  • [Proxy] Do not preserve host when forwarding admin requests #16342
  • [Proxy] Switch to rely on Netty for Hostname Verification #15824
  • [Proxy] Support zero-copy of NIC to NIC on Proxy #15678
  • [Proxy] Suggestion put brokerProxyAllowedTargetPorts in proxy.conf #15069
  • [Proxy] Exit if proxy service fails to start #15076
  • [Proxy] Add support of PrometheusRawMetricsProvider for the Pulsar-Proxy #14681
  • [Proxy] Fail proxy startup if brokerServiceURL is missing scheme #14682
  • [Proxy] Fix DNS server denial-of-service issue when DNS entry expires #15403
  • [Proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler #15415
  • [WebSocket] Support encryption in websocket proxy #16234
  • [WebSocket] Fix MultiTopicReader#getConsumer ClassCastException #15534
  • [WebSocket] Add KeyStore support in WebSocket, Function Worker HTTPS Servers #15084
  • [WebSocket] Fix ClassCastException when user create MultiTopicReader #14316
  • [Package Management] Fix the new path /data introduced regression #15367
  • [Schema] Add decode InputStream for Schema #16659
  • [Schema] Expose timestamp field for SchemaData&SchemaInfo #16380
  • [Schema] Add classLoader field for SchemaDefinition #15915
  • [Schema] Fix conversion of TimestampMillisConversion has no effect when Jsr310Conversion enabled #15863
  • [Schema] Ensure the deletion consistency of topic and schema #14608
  • [Tiered Storage] Shared subscription: improvement with offloaded ledgers #16417
  • [Tiered Storage] Fix the wrong secret key name get from env #15814
  • [Tiered Storage] Add pure S3 provider for the offloader #15710
  • [Tiered Storage] Autorecovery default reppDnsResolverClass to ZkBookieRackAffinityMapping #15640
  • [Tiered Storage] Reduce CPU usage when offloading ledgers #15063
  • [Tiered Storage] Fix potential NPE in MockManagedLedger #15006
  • [Tiered Storage] Add API to scan objects on Tiered Storage #14930
  • [Tiered Storage] Fix incorrect entryId in warn log when reading entry from tiered storage #14685
  • Include Pulsar shell in the released artifacts #18583
  • Add lombok plugin to solve Javadoc issue #18520
  • Remove cmd-line level test retries #16524
  • Fix potentially incompatible Java opts caused by IS_JAVA_8 #15444
  • Remove obsolete jdk11 and errorprone-jdk11 maven profiles #15505
  • Remove internal dependency: pulsar-functions-instance #14925
  • Enable retries for apt-get when building Pulsar docker image #14513
  • Migrate Docker images and CI to Java 17 #14355

Library updates​

  • Remove versions that are handled by netty-bom #18629
  • Upgrade Netty to 4.1.86.Final and Netty Tcnative to 2.0.54.Final #18599
  • Upgrade jackson-databind to 2.13.4.2 to get rid of CVE-2022-42003 #18394
  • Bump bookkeeper version to 4.15.3 #18455
  • Upgrade jackson-databind to 2.13.4.2 to get rid of CVE-2022-42003 #18394
  • Upgrade protobuf to 3.19.6 to get rid of CVE-2022-3171 #18086
  • Bump commons-text to 1.10.0 fix CVE-2022-42889 #18053
  • File tiered storage: upgrade jettison to get rid of CVE-2022-40149 #18022
  • Upgrade scala-library to get rid of CVE-2022-36944 #18021
  • Upgrade JacksonXML to 2.13.4 #18020
  • Upgrade to Jetty to 9.4.48.v20220622 #16520
  • Bump maven-dependency-plugin to 3.3.0 #16318
  • Bump os-maven-plugin version from 1.4.1.Final to 1.7.0 #16308
  • Bump presto.version to run PrestoServer on JDK11+ #16163
  • Update fastjson version to 1.2.83 #16148
  • Bump PyYAML from 5.3.1 to 5.4.1 to solve CVE-2020-14343 #15989
  • Bump spring version to 5.3.20 to solve CVE-2022-22970 #15699
  • Bump snappy zstd version to fix CompressorCodecBackwardCompatTest on Apple M1 #15698
  • Athenz: do not use uber-jar and bump to 1.10.50 to remove jackson-databind shaded dependency #14884
  • Remove --illegal-access errors resulting from Google Guice - Pulsar IO, Offloaders and Pulsar SQL - Bump Guice to 5.1.0 #14300
  • Bump prometheus client version from 0.5.0 to 0.15.0 #13785
  • Upgrade log4j2 version to 2.18.0 #16884

Documentation​

This section only highlights the availability of docs serving existing features.

For the comprehensive list of doc-related improvements in 2.11.0, you can go to GitHub.

A special thanks to the following contributors (alphabetic Github IDs) who helped add the docs for Pulsar 2.11.0.

0xflotus, 315157973, 704237006, AlphaWang, alpreu, AlvaroStream, AnonHxy,Anonymitaet, asafm, cbornet, codelipenghui, coderzc, D-2-Ed, DaveDuggins, Demogorgon314, ericsyh, fantapsody, futeng, hangc0276, heesung-sn, hrsakai, HQebupt, horizonzy, Huanli-Meng, gaozhangmin, ikilobyte, Jason918, komalatammal, labuladong, larshp, lgxbslgx, lhotari, liangyepianzhou, lin-zhao, liudezhi2098, liuzhuang2017, Mans2singh, massakam, mattisonchao, maxsxu, mendonk, merlimat, michaeljmarshall, misselvexu, momo-jun, MTwz, nahguam,nicoloboschi, nodece, poorbarcode, Pradhyumnakashyap, rdhabalia, RobertIndie, samredai, Shawyeok, Sherlock113, shibd, Shoothzj, SignorMercurio, Technoboy-, tjiuming, tisonkun, urfreespace, wangjialing218, wenbingshen, youzipi, and zwOvO.

Was this helpful?