Skip to main content

Apache Pulsar 2.8.0

2021-06-12​

Update notice​

Due to a breaking change in the Schema API, it may happen that some Pulsar Functions or Pulsar IO connectors fail to work, throwing an IncompatibleClassChangeError.

In this case, you need to rebuild your Function using Apache Pulsar 2.8.0 as a dependency and redeploy it.

If you are running on Kubernetes, you can temporarily let the Functions Worker pod run with a previous version of Pulsar in order to not cause downtime.

For more context about this issue, see [Pulsar Functions and IO] Cannot upgrade Function built for Pulsar 2.7 to Pulsar 2.8.

PIPs​

Transactions​

  • Support pending ack state persistence #8881
  • Stable position and low watermark for the transaction buffer #9195
  • Transaction timeout support #9229
  • Transaction coordinator retry to complete the transaction #9236
  • Fix race condition when appending transaction log #9238
  • Transaction buffer snapshot #9490
  • Add metrics for transaction coordinator #9706
  • Clean the useless transaction individual acknowledgements based on low watermark #9722
  • Fix memory leak when deleting transaction marker #9751
  • Check the transaction state at the client side #9776
  • Clean aborted transactions for the transaction buffer #9974
  • Fix transaction coordinator retry to end transaction #10131
  • Fix NPE when opening a new transaction #10139
  • Fix transaction log failed to recover #10146
  • Fix transaction coordinator recover timeout #10162
  • Handling committing and aborting state when recovering transaction coordinator #10179
  • Fix NPE in case of enableTransactionCoordinator=false #10182
  • Fix transaction buffer client timeout #10206
  • Fix recover max local id issue for the transaction coordinator #10224
  • Support using transactions on standalone #10238
  • Fix transaction buffer lookup issue #10257
  • Fix transaction timeout issue at the client side #10323
  • Fix transaction client reconnect issue after transaction coordinator unloaded #10327
  • Fix transaction timeout not canceled after expired #10366
  • Transaction log low watermark optimization #10422
  • Fix the issue of transaction buffer client channel is inactive #10407
  • Add Admin API for getting transaction coordinator stats #10639
  • Add Admin API for getting transaction in transaction buffer stats #10642
  • Add Admin API for getting transaction in pending ack stats #10648
  • Add Admin API for getting transaction buffer stats and pending ack stats #10650
  • Add Admin API for getting transaction coordinator internal stats #10653
  • Setup transaction metadata with metadata store #10677
  • Fix issue with acknowledge messages from multiple subscriptions of a topic #10689
  • Admin API for getting transaction metadata #10690
  • Admin API for getting slow transactions #10701
  • Fix transaction log handle managed ledger WriteFail state #10711
  • Admin API for getting pending ack internal stats #10725
  • Fix transaction ack delete marker position when don't have transaction ack #10741
  • Fix transaction pending ack generate managedLedgerStats fail exception #10797
  • Use Zookeeper Prometheus metric provider to export Zookeeper metrics #10803
  • Always allow system topic for transaction buffer snapshot auto create #10876

Security​

  • Optional auth method name header in HTTP authentication #6799
  • Topics level authorization support #7523
  • Authorization for setting topic/subscription auto-creation and subscription expire time on namespace #7900
  • Allow serializable stream-provider field into AuthenticationTls #10020
  • Add configuration for running OWASP Dependency Check for all modules #10288
  • Update default TLS protocols to TLSv1.3 and TLSv1.2 for the broker and proxy #10598

Broker​

  • Share EventLoopGroup between broker and BookKeeper client #2603
  • Dispatch batch messages according to consumer permits 7266
  • Improve the max pending publish buffer mechanism 7406
  • Allow disabling HTTP TRACE/TRACK verbs #7907
  • Fix DispatchRateLimiter does not take effect #8611
  • Validate namespace isolation policy regex before updating #8804
  • Perform automatically cert refresh for pulsar-admin #8831
  • Fix updating rack info dynamically #8844
  • Fix deadlock when checking message expiration #8877
  • Support limit max topics per namespace #8942
  • Make ledger rollover check task internally #8946
  • Clean up topic that failed to unload from the cache #8968
  • Support get broker response for the message acknowledgement #8996
  • Support message index for a topic #9039
  • Supporting limit max topics per namespace by namespace policy #9042
  • Streaming dipsatcher support #9056
  • Make Netty acceptor threadPool size configurable #9061
  • Fix deadlock when unloading namespace bundles #9116
  • Fixed checking for maxTopicsPerNamespace #9121
  • Change the service URL to a not required param when creating a cluster #9127
  • Support setting replicator dispatch rate policy at the topic level #9175
  • Fix max topic for namespace does not work #9193
  • Fix memory leak of the managed ledger interceptor #9194
  • Fix max consumers per topic cannot be disabled at the namespace level #9214
  • Support schema REST API for V1 topics #9218
  • Fix peek message metadata while enabling broker entry metadata #9255
  • Support fetching metadata from entry data in publish callback #9257
  • Wait for the async broker port listener close operations to complete at shutdown #9308
  • Support script based mapping network topology #9363
  • Make managed ledger storage configurable #9397
  • Support setting enabled subscription types #9401
  • Fixed NPE and cache invalidation in leader election #9460
  • Fix exception when get an optional field for Protobuf message #9468
  • Ignore replicated subscription configurations from the client when disabled by broker #9523
  • Support expose producer metrics through Prometheus endpoint #9541
  • Fix NPE that occurs in PersistentStickyKeyDispatcherMultipleConsumers when debug log enabled #9587
  • Make LocalPolicies immutable to avoid concurrent modify inconsistent #9598
  • Fix writing/encoding issue of GenericJsonRecord #9608
  • Expose the native record for struct schema #9614
  • Add metrics for producer throttling #9649
  • Fix MaxUnackedMessagesPerConsumer cannot be changed dynamically and cannot be disabled #9657
  • Added support for force deleting tenant #9677
  • Fix managed ledger not found exception when force delete namespace #9691
  • Reduce CPU consumption of metrics creation #9735
  • Ensure read-lock is not continuously held on a section while iterating over concurrent maps #9787
  • Add pending read subscription metrics to stats-internal #9788
  • Allow broker to discover and unblock stuck subscription #9789
  • Disabled the tenants/namespaces force deletion by default #9819
  • Add metrics for the connections of the broker #9876
  • Make readModifyUpdate in MetadataCacheImpl thread-safe #9900
  • Optimize NamespaceBundle.toString() which is on the hot path #9976
  • Support set compaction threshold in broker.conf #9989
  • Support set properties for a namespace #10015
  • Fix cannot cleanup expired data after managed-ledger restart #10087
  • [Default configuration] Enable sticky read by default #10090
  • Add time based backlog quota #10093
  • Fix IllegalStateException in PersistentReplicator #10098
  • Support set max consumers per subscription for non-persistent dispatcher #10121
  • Limit the number of producers/consumers that can connect per topic for each IP address #10188
  • Close namespace clients when PulsarService is closed #10196
  • Shutdown Broker gracefully, but forcefully after brokerShutdownTimeoutMs #10199
  • Update the authentication data when an authentication refresh happens #10203
  • Add preciseTopicPublishRateLimiterEnable to broker.conf #10216
  • Fix the typo in the chunkedMessageRate stats and metrics #10223
  • Fix entry cache size to be a negative value #10225
  • Fix replicated subscriptions related LightProto issues #10247
  • Pause the replicated subscriptions snapshots when there is no real traffic #10292
  • Fix the inconsistency of advertised address #10312
  • Support listenerName for HttpLookupService #10319
  • Support truncate topic #10326
  • Fix authorization error if partition number of partitioned topic is updated 10333
  • Fix managed ledger name that transaction log used #10334
  • Catch topic policy not hit exception in handleSubscribe #10341
  • Fix ConcurrentModificationException when attempting to update local broker data #10347
  • Support seek to separate messageId/time for multiple topic reader #10348
  • Resource locks should automatically revalidate after a metadata session is re-established #10351
  • Add authentication data for the remote cluster #10357
  • Support array type claims in JWT #10375
  • Optimize locks in AuthenticationAthenz #10381
  • Prevent carrying state of PositionImplRecyclable when recycled #10404
  • Dispatch messages to consumer with permits #10417
  • Fix NPE in unblock stuck subscription task when dispatcher has not created #10430
  • Fix topic loading fails without any error when replicator init fails #10432
  • Set timeout to unblock web-threads on update partition API #10447
  • Fix CPU 100% when deleting namespace #10454
  • Remove residual info after forcibly deleting the namespace #10465
  • Fix potential memory leak of TopicPolicies #10466
  • Fix publish callback's entry data is null during ledger rollover #10467
  • Fix readModifyUpdate should return the new value in the future for metadata cache #10474
  • Fix partitioned system topic check issue #10529
  • Removed AspectJ based metrics for ZooKeeper #10533
  • Allow creating MetadataCache with custom serde #10543
  • Fix ack receipt version check issue #10551
  • Expose average message size metrics for a topic #10553
  • Fixed missed ZK caching when fetching list of namespaces for a tenant #10594
  • Setup pulsar cluster with MetadataStore #10600
  • Setup initial namespaces with MetadataStore #10612
  • Convert bundle split into an HTTP async operation #10619
  • Add metrics for non-contiguous deleted messages range #10638
  • Fix consumer related topic stats only available while consumer or reader are connected #10644
  • Allow configuring the number of BK client worker threads #10649
  • Fix ConcurrentOpenLongPairRangeSet remove all ranges #10656
  • Ensure all the ReadHandle gets properly closed on cache invalidation #10659
  • Avoid the context switch when managedLedgerNewEntriesCheckDelayInMillis=0 #10660
  • Allow configuring busy-wait in broker and client #10661
  • Use a single JWT parser instance #10664
  • Fix issues in advanceNonDurableCursors #10667
  • LockManager use a concurrent hash map to handle locks notifications #10680
  • Avoid making copies of internal maps when iterating #10691
  • Fix issue where StackOverflowError occurs when trying to redeliver a large number of already acked messages #10696
  • Fix NPE when filtering read entries #10704
  • Fix deadlock when enabling managed ledger interceptor #10706
  • Fix inconsistent behavior in LongPairRangeSet #10713
  • Fix repeated iterator generation of LongPairRangeSet #10722
  • Cancel scheduled tasks as the first step in closing for the managed ledger #10739
  • Prevent race conditions between timeout and completion of managed ledger #10740
  • Add advertised listener name for geo-replicator #10751
  • Add a read lock when traversing batchDeletedIndexes #10763
  • Replace use of Prometheus client CollectorRegistry.getSampleValue #10764
  • Release OpAddEntry.data when entry is copied and discarded for the managed ledger #10773
  • Avoid warning logs on topic policies not present #10785
  • Remove pulsar-zookeeper module and call zookeeper quorumPeerMain to start zookeeper #10823
  • Fix consumer stuck issue due to reuse entry wrapper #10824
  • Fix possible data race in getFirstAvailableConsumerPermits #10831

Functions​

  • Update default function log location in function_worker.yaml #8470
  • Optimize batch source discovery and task ack #8498
  • Provide an interface for functions worker service #8560
  • Improve naming of threads used in batch source #8608
  • Fix the reader used to read the metadata topic during worker service initialization not been closed #8637
  • Add timeout to hasMessageAvailable to leader election process #8687
  • Support key based batch builder for Go Functions #8761
  • Fix panic when discarding message for Go Functions #8776
  • Move initialize dlog namespace metadata to bin/pulsar #8781
  • Support URL fetching for Go/Python Functions #8808
  • Add cmd flag for retaining key ordering #8886
  • Supports to use package command #8973
  • Support create and update sink with package name #8987
  • Support create and update source with package name #8988
  • Allow stats operations not to be blocked in functions #9005
  • Adding timeout to open table call for function state #9006
  • Fix get function info error of REST API #9115
  • Improve the error message when creating sinks #9131
  • Expose Pulsar Admin through Function context #9246
  • Enable Function Workers to use exclusive producer to write to internal topics #9275
  • Add additional logging when setting up state table for function instance #9304
  • Allow memory limit to be set for the pulsar client used in the ThreadRuntime in Pulsar Function #9320
  • Make admin operations on Statestore non blocking #9348
  • Fix maxPendingAsyncRequests not affect Kubernetes Runtime #9349
  • Add download directory support to function Kubernetes Runtime #9377
  • Optimize built-in source/sink startup by eliminating redundant NAR unpacking and checksum calculation #9413 #9500
  • Enhance Kubernetes manifest customizer with default options #9445
  • Fix possible deadlock on broker-function service startup #9499
  • Close InputStreams properly #9568
  • Add maximum allowed amount of resources setting for functions #9584
  • Support writing general records to Pulsar sink #9590
  • Move metrics port configuration to InstanceConfig #9610
  • Add resource granularity settings for functions #9736
  • Prevent invalid broker or proxy configuration for authorization #9746
  • Log stacktraces of threads that failed to terminate on shutdown within timeout in ExecutorProvider #9840
  • Support get partition index for a Record #9947
  • Deleting a Pulsar Function with a name that includes a colon character crashes the pulsar broker #9946
  • Exposing Prometheus metrics for Pulsar function local run mode #10156
  • Expose local run config metrics-port-start to CLI #10185
  • Fix Pulsar Function localrun with multiple instances and metrics server is enabled #10208
  • Improve localrun performance by using JVM options #10273
  • Fix function API can not use authdata to check superuser #10364
  • Fix potential bug getting stats and remove unnecessary error log #10500
  • Fix deadlock on Monitoring thread blocked by LeaderService.isLeader() #10502
  • Allow user to set custom configs to plugin worker service #10504
  • K8s Function Name Length Check Allows Invalid StatefulSet #10531
  • Get function cluster from broker config when start function worker with broker #10552
  • Process async results in the same Java runnable thread #10618
  • Support using AutoProduceBytesSchema as the function output schema #10716
  • ReflectionUtils use Class.forName in order to properly discover classes in Functions Runtime while using DefaultImplementation #10827
  • Fix the out of index issue when dispatch messages based on the avgBatchSizePerMsg #10828

IO Connectors​

  • [Kafka Source Connector] Fix invalid topic name generation #9035
  • [Kafka Source Connector] Allow managing Avro encoded messages #9448
  • Configure Pulsar IO connector YAML file generator for all connectors #9629
  • [Kafka Source Connector] KeyValue schema support for KafkaBytesSource #10002
  • Allow using GenericObject for developing a Sink connector #10034
  • Sink<GenericObject> unwrap internal AutoConsumeSchema and allow to handle topics with KeyValue schema #10211
  • Using ObjectMapper instead of Gson to parse Source/SInk configuration #10441
  • Expose subscription type in the SinkContext #10446
  • Add the ability to seek/pause/resume for a topic to the SinkContext #10498
  • Use Message.getReaderSchema() in Pulsar IO Sinks when possible #10557
  • [Kinesis Sink Connector] Fix backoff class not found #10744
  • [Kinesis Sink Connector] Fix does not acknowledge messages #10769
  • [Kafka Sink Adaptor] Support non-primitive schemas #10410

Pulsar SQL​

  • Fix the misleading setting in presto configuration #8549
  • Fix injection factory cast error #9472
  • Add max split entry queue size bytes limitation #9628
  • Fix Pulsar SQL query bytes schema data error #9631
  • Fix Pulsar SQL query compression data #9663
  • Support native Protobuf decoder #9841
  • Support query uppercase topic name #9980
  • Only return persistent topic when list tables #10368
  • Fix Presto startup on JDK11 #10559
  • Fix pulsar sql issues when run select count(*) for the table with primary schema #10840

Tiered Storage​

  • Fix AWS credentials usages #8950
  • Support ALI OSS tiered storage #8985

Proxy​

  • Fix the proxy does not support partitioned topic auto-creation type #7903
  • Enable Conscrypt for Jetty in the Broker and in the Proxy #10541

Pulsar Admin​

  • Support get applied message TTL policy for a topic #9225
  • Support get applied inactive topic policy for a topic #9230
  • Support get applied delayed delivery policy for a topic #9245
  • Support get applied max unacked message per subscription policy for a topic #9290
  • Support get applied max producer for a topic #9293
  • Support get applied max consumer for a topic #9296
  • Support get applied deduplication status policy for a topic #9339
  • Support get applied retention policy for a topic #9362
  • Support get applied offloader policy for a topic #9505
  • Support get applied max unacked messages per consumer for a topic #9694
  • Support get applied dispatch rate policy for a topic #9824
  • Support get applied subscription dispatch rate policy for a topic #9827
  • Support get applied backlog quota policy for a topic #9828
  • Support get applied persistence policy for a topic #9831
  • Support get applied cluster subscribe rate for a topic #9832
  • Support get applied replicator dispatch rate for a topic #9833
  • Support get applied compaction threshold #10038
  • Lazily init PulsarAdmin in PulsarAdminTool #9312
  • Fix create partition of existing topic does not throw RestException #9342
  • Support get service URL of the leader broker #9799
  • Support get persistent topics or non-persistent topics for pulsar admin client #9877
  • Fix can not disable and remove max consumer per subscription #10070
  • Add support for setting time based limit on backlog quota #10401
  • Make client-admin-api to use interfaces with builders instead of POJOs #10818

Tools​

  • [Pulsar Perf] Support load WebSocket service URL from config file #9000)
  • [BookKeeper Shell] Make bookkeeper shell more user friendly #9281
  • [Client Tools] Supports end-to-end encryption #9615
  • [Pulsar Perf] Support specify topics and subscriptions #9716
  • [Client Tools] Allow printing GenericRecord contents #9785
  • [Pulsar Perf] Fix compatibility issues with previous version #9838
  • [Pulsar Perf] Add ability to create partitioned topics #9859
  • [Client Tools] handle data with different schemas when consuming messages #10301
  • [Client Tools] Fix NoClassDefFoundError when running pulsar cli tools in dev directory #10807

Dependencies​

  • Upgrade Kubernetes client and remove sundr-codegen #8576
  • Upgrade ZooKeeper to 3.6.2 and Curator to 5.1.0 #8549
  • [CVE-2020-26238] Upgrade cron-utils to 9.1.3 and sprint-context to 5.3.1 #8822
  • Upgrade Swagger to 1.6.2 and Maven Swagger Plugin to 3.1.7 #8845
  • Upgrade to Apache Avro 1.10.2 #9898
  • Add jersey-client as dependency of pulsar-client-auth-sasl #10055
  • Upgrade Athenz version to 1.10.9 and remove yahoo.bintray.com repository #10079
  • [CVE-2020-15250] Upgrade junit version to 4.13.1 #10147
  • [CVE-2020-8908,CVE-2018-10237] Upgrade jclouds to 2.3.0 #10149
  • Remove jackson-mapper-asl dependency to resolve multiple CVEs #10262
  • [CVE-2021-21409] Upgrade Netty to 4.1.63.Final #10266
  • [CVE-2018-12541] Upgrade vertx to 3.9.7 #10261
  • Upgrade BookKeeper to 4.14.1 #10686
  • [CVE-2021-28169] Upgrade Jetty to 9.4.42.v20210604 #10907
Was this helpful?