Skip to main content

Apache Pulsar 3.0.2

2023-12-01​

Broker​

  • [fix][broker] Fix namespace bundle stuck in unloading status (21445) #21565
  • [fix][broker] Fix create topic with different auto creation strategies causes race condition #21545
  • [fix][broker] Fix thousands orphan PersistentTopic caused OOM #21540
  • [fix][broker] Fix failure while creating non-durable cursor with inactive managed-ledger #21508
  • [fix][broker] Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0 #21496
  • [fix][broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider #21486
  • [fix][broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy #21481
  • [fix][broker] Fix namespace bundle stuck in unloading status #21445
  • [fix][broker] Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered. #21434
  • [fix][broker] Fix heartbeat namespace create event topic and cannot delete heartbeat topic #21360
  • [fix][broker] Fix heartbeat namespace create transaction internal topic #21348
  • [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager #21332
  • [fix][broker][branch-3.0] Fix lookup heartbeat and sla namespace bundle when using extensible load manager (21213) #21313
  • [improve][broker] use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error #21282
  • [fix][broker][branch-3.0] Fix inconsistent topic policy #21256
  • [fix][broker]Fixed produce and consume when anonymousUserRole enabled #21237
  • [fix][broker] Fix inconsistent topic policy #21231
  • [fix][broker] Fixed reset for AggregatedNamespaceStats #21225
  • [fix][broker] Fix lookup heartbeat and sla namespace bundle when using extensible load manager #21213
  • [improve][broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex #21211
  • [fix][broker] replicator leak when removeReplicator in NonPersistentTopic #21205
  • [improve][broker] Print warn log if ssl handshake error & print ledger id when switch ledger #21201
  • [fix][broker] Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21179
  • [improve][broker] improve read entry error log for troubleshooting #21169
  • [fix][broker][branch-3.0] Fix write duplicate entries into the compacted ledger after RawReader reconnects (21081) #21165
  • [improve][broker][branch-3.0] Make read compacted entries support maxReadSizeBytes limitation (21065) #21164
  • [fix][broker] Avoid splitting one batch message into two entries in StrategicTwoPhaseCompactor #21156
  • [fix][broker] Make specified producer could override the previous one #21155
  • [improve][broker] Upgrade bookkeeper to 4.16.3 #21146
  • [improve][broker] Improve logs for troubleshooting #21141
  • [fix][broker] Fix web tls url null cause NPE #21137
  • [fix][broker] Fix unack count when mixing non batch index and batch index acks #21126
  • [fix][broker] Fix unsubscribe non-durable subscription error #21099
  • [fix][broker] Fix isolated group not work problem. #21096
  • [fix][broker] Fix deleting topic not delete the related topic policy and schema. #21093
  • [fix][broker] Avoid splitting one batch message into two entries in StrategicTwoPhaseCompactor #21091
  • [fix][broker] revert remove duplicate topics name when deleteNamespace #21087
  • [fix][broker] Cleanup correctly heartbeat bundle ownership when handling broker deletion event #21083
  • [fix][broker] Fix write duplicate entries into the compacted ledger after RawReader reconnects #21081
  • [fix][broker] remove bundle-data in local metadata store. #21078
  • [fix][broker] Make sure all inflight writes have finished before completion of compaction #21067
  • [improve][broker] Make read compacted entries support maxReadSizeBytes limitation #21065
  • [improve][broker] Improve cache handling for partitioned topic metadata when doing lookup #21063
  • [fix][broker] Use MessageDigest.isEqual when comparing digests #21061
  • [fix][broker] Fix potential case cause retention policy not working on topic level #21041
  • [fix][broker] fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21025
  • [fix][broker] Fix incorrect unack msk count when dup ack a message #20990
  • [fix][broker] Fix can't stop phase-two of compaction even though messageId read reaches lastReadId #20988
  • [fix][broker] Fix compaction subscription delete by inactive subscription check. #20983
  • [fix][broker] Fix message loss during topic compaction #20980
  • [fix][broker] Fix incorrect number of read compacted entries #20978
  • [fix][broker] Producer is blocked on creation because backlog exceeded on topic, when dedup is enabled and no producer is there #20951
  • [fix][broker]Fix chunked messages will be filtered by duplicating #20948
  • [fix][broker] rackaware policy is ineffective when delete zk rack info after bkclient initialize #20944
  • [fix][broker]Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin #20939
  • [fix][broker] fix MessageDeduplication throw NPE when enable broker dedup and set namespace disable deduplication. #20905
  • [fix][broker] In replication scenario, remote consumer could not be registered if there has no message was sent #20888
  • [fix][broker] Fix inconsensus namespace policies by getPoliciesIfCached #20855
  • [improve][broker] Avoid print redirect exception log when get list from bundle #20846
  • [fix][broker] Inconsistent behaviour for topic auto_creation #20843
  • [fix][broker] Can not receive any messages after switch to standby cluster #20767
  • [fix][broker] Fix get topic policies as null during clean cache #20763
  • [fix][broker] Avoid throwing RestException in BrokerService #20761
  • [fix][broker] Gracefully shutdown does not work with admin cli in standalone #20709
  • [fix][broker] Fix ModularLoadManagerImpl always delete active bundle-data. sec ver. #20620
  • [improve][broker] Add consumer-id into the log when doing subscribe. #20568
  • [fix][broker] consider iowait as idle. #19110
  • [fix][broker] fix bug caused by optimistic locking #18390
  • [fix][ml] Fix orphan scheduled task for ledger create timeout check #21542
  • [fix][ml] Fix unfinished callback when deleting managed ledger #21530
  • [fix][ml] fix discontinuous ledger deletion #20898
  • [fix][ml] Fix thread safe issue with RangeCache.put and RangeCache.clear #21302
  • [fix][ml] fix wrong msg backlog of non-durable cursor after trim ledgers #21250
  • [fix][ml] Reader can set read-pos to a deleted ledger #21248
  • [improve][ml] Persist mark deleted ops to ZK if create cursor ledger was failed #20935
  • [fix][txn] OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly #21505

Clients​

  • [fix][client] Fix print error log 'Auto getting partitions failed' when expend partition. #21485
  • [fix][client] fix reader.hasMessageAvailable return false when incoming queue is not empty #21259
  • [improve][client] Merge lookup requests for the same topic #21232
  • [fix][client] fix same producer/consumer use more than one connection per broker #21144
  • [fix][client] Fix repeat consume when using n-ack and batched messages #21116
  • [fix][client] Avoid ack hole for chunk message #21101
  • [fix][client] Fix logging problem in pulsar client #21094
  • [fix][client] Fix consumer can't consume resent chunked messages #21070
  • [fix][client] Fix cannot retry chunk messages and send to DLQ #21048
  • [fix][client] Fix RawReader hasMessageAvailable returns true when no messages #21032
  • [fix][client] Fix perf-producer get OOM with high publish latency #20914
  • [improve][client] Disable polling pattern topics when TopicListWatcher is enabled. #20779
  • [fix][client] Fix subscribing pattern topics through Proxy not working #20739
  • [fix][client] Messages lost when consumer reconnect #20695
  • [fix][txn] fix the consumer stuck due to deduplicated messages in pending ack state #21177

Pulsar IO and Pulsar Functions​

  • [fix][fn] fix functions_log4j2.xml delete strategy config #21215
  • [fix][fn] Fix ProducerConfig cannot update error #21037
  • [fix][fn] Fix the --batch-builder not working error for functions #21023
  • [fix][io] Fix --retain[-key]-ordering not working error for sink #21060
  • [fix][io] Update test certs for Elasticsearch #21001
  • [fix][io][branch-3.0] Not restart instance when kafka source poll exception. #20818
  • [fix][io] elastic-search sink connector not support JSON.String schema. #20741
  • [feat][sql] Support UUID for json and avro #21267
  • [improve][sql] Fix the wrong format of the logs #20907

Observability​

CLI​

  • [fix][admin] Fix get topic stats fail if a subscription catch up concurrently #20971

Others​

  • [fix][bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem. #21327
  • [fix][build] Fix apt download issue in building the docker image #21489
  • [fix][build][branch-3.0] Fix in-correct license definetion #21435
  • [fix][test][branch-3.0] Fix AutoRecovery flaky test. #21418
  • [fix][test] Fix LocalBookkeeperEnsemble resource leak in tests #21407
  • [improve][ci] Add new CI unit test group "Broker Group 4" with cluster migration tests #21391
  • [fix][test] Fix a resource leak in ClusterMigrationTest #21366
  • [fix][ci] Fix docker image building by releasing more disk space before building #21365
  • [fix][sec] Fix MultiRoles token provider when using anonymous clients #21338
  • [fix][test] Fix flaky test NarUnpackerTest #21328
  • [fix][auto-recovery][branch-3.0] Fix PulsarLedgerUnderreplicationManager notify problem. #21312
  • [fix][metadata] Fix zookeeper related flacky test #21310
  • [fix][sec] Add OWASP Dependency Check suppressions #21281
  • [improve][proxy] Not close the socket if lookup failed caused by too many requests #21216
  • [improve][auto-recovery] Migrate the replication testing from BookKeeper to Pulsar. #21188
  • [fix][auto-recovery] Fix pulsar ledger auditor dead lock problem. #21181
  • [fix][auto-recovery] Fix PulsarLedgerUnderreplicationManager notify problem. #21161
  • [fix][bk] Improve to the ReplicaitonWorker performance by deleting invalid underreplication nodes #21160
  • [fix][auto-recovery] Fix metadata store deadlock due to BookkeeperInternalCallbacks.Processor #21159
  • [fix][bookie] Fix RocksDB configuration #21157
  • [fix][proxy] Fix Proxy 502 gateway error when it is configured with Keystore TLS and admin API is called #21077
  • [fix][auto-recovery] Improve to the ReplicaitonWorker performance by deleting invalid underreplication nodes #21059
  • [fix][bk] Correctct the bookie info after ZK client is reconnected #21035
  • [improve][proxy] Support disabling metrics endpoint #21031
  • [fix][meta] Fix deadlock in AutoRecovery. #21010
  • [improve][ws] add cryptoKeyReaderFactoryClassName into the file websocket.conf #20840
  • [improve][offload] Extend the offload policies to allow specifying more conf #20804
  • [improve][offload] Create offload resources lazily #20775
  • [improve][offload] Support to configure more offload driver #20736
  • [fix][test] Close the resource after the test #20778
  • [fix][schema] Only handle exception when there has #20730
  • [improve][admin] Remove duplicate topics name when deleteNamespace #20683
  • [fix][txn] fix consumer can receive aborted txn message when readType is replay #19815

Library updates​

  • Upgrade Zookeeper to 3.8.3 to address CVE-2023-44981 #21398
  • Upgrade Netty to 4.1.100 to address CVE-2023-44487 #21397
  • Upgrade Jetty to 9.4.53 to address CVE-2023-44487 #21395
  • Upgrade Guava to 32.1.1 to address CVE-2023-2976 #20699
  • Upgrade Netty to 4.1.94.Final to address CVE-2023-34462 #20632
  • Upgrade to jetcd to 0.7.5 #20339
  • Upgrade Lombok to 1.18.30 to support compiling with JDK21 #21278
  • Bump avro version to 1.11.3 for CVE-2023-39410 #21341
  • Bump GRPC version to 1.55.3 to fix CVE #21057
  • Bump broker okio version to 3.4.0 #21064

For the complete list, check the Pulsar 3.0.2 changelog.

Was this helpful?