Apache Pulsar Release Notes (Legacy)

2.4.2

2019-12-04

Fixes

Fixed don't set interrupt flag again after catching interrupt exception in Pulsar Client #5643
Fixed data is not deleted after expiration due to connected readers #5621
Fixed the go client docs missing in website #5595
Fixed schema def build error with protobuf schema #5569
Fixed docs about reset cursor #5551
Fixed repeated initialization of connectorsManager #5545
Fixed Functions unnecessarily restart during FunctionRuntimeManager init phase #5527
Fixed list non-persistent topics shows the persistent topics #5502
Return after triggering callback with empty result #5500
Fixed dispatcher skipping delivery of a batch during concurrent replays #5499
Fixed reader_listener option for Python API #5487
Fixed wrongly report "3600 messages have timed-out" #5477
Fixed broken custom auth-provider that uses authenticationData #5462
Fixed negative ack tracker constructor sequence #5453
Fixed StringSchema static initialization #5445
Fixed message corruption on OOM for batch messages #5443
Fixed couple functions related integration tests #5434
Fixed bug that namespace policies does not take effect due to NPE #5408
Fixed race condition : Failed to read-more entries on dispatcher #5391
Fixed potential deadlock that can occur in addConsumer #5371
Fixed proxy to be able to re-send request body #5361
Fixed pulsar can't load the customized SerDe #5357
Fixed instability in Pulsar Function window integration test #5337
Fixed bk write failure, use signals to resume writing #5322
Fixed bad_weak_ptr error when closing producer #5315
Fixed typo(massage->message) causing prometheus metrics display error #5299
Fixed invalidate cache on zk-cache timeout #5298
Fixed memory leak caused by not being executed ClientConnection destructor #5286
Fixed producer blocked after send an over size message while batch enabled #5282
Fixed race condition while triggering message redelivery after an ack-timeout event #5276
Fixed behavior when getting a key from functions state that doesn't exist #5272
Fixed Cmake to build _pulsar for osx #5263
Fixed client backoff setting does not take effect #5261
Fixed memory leak caused by deadline_timer holding object reference #5246
Fixed in Message Deduplication that may cause incorrect client/broker interaction #5243
Fixed bug that fails to search namespace bundle due to NPE #5191
Fixed bug that message delivery stops after resetting cursor for failover subscription #5185
Fixed exception type check order bug #5174
Fixed spark receiver to account for all the consumer config options #5152
Fixed broker fails to start with function worker enabled and broker client using TLS #5151
Fixed deadlock when resetting cursor #5136
Fixed windowed functions were broken when we changed java function instance to use classloaders #5117
Fixed storage size always 0 without subscription #5108

Enhancements

Add subscribe position param for consumer of sink #5532
Efficiency improvements for delay delivery tracker #5498
Add is_read_compacted to create_reader() in python API #5483
Make some common use method of ManagedLedger public #5472
Avoid leak on publish failure on batch message #5442
Record message failure and avoid exiting from process on publish failure #5441
Add support for partitioned topic consumer seek by time #5435
Add default loader for latest pyyaml #5432
Trim messages which less than mark delete position for message redelivery #5378
Make skip all messages async #5375
Set default ensemble size to 2 in service conf, to match broker.conf #5339
Only seek when reading unexpected entry #5356
Don't require both region and endpoint to be specified #5355
If cursor is not durable, close dispatcher when all consumers are removed from subscription #5340
Disable stickyRead by default #5321
Allow to specify delivery delay in C++ client #5317
Add debug log + fix thread-factory name #5302
Don't attempt to append on read-only cursor ledger #5297
Close previous dispatcher when subscription type changes #5288
Improve error handling logic for effectively once #5271
Upgrade dependencies for security fixes #5232
Ensure consumer background tasks are cancelled after subscribe failures #5204
Added missing enum value KeyShared on the python wrapper #5196
Make some member variables of Dispatcher volatile #5193
Ensure getting list of topics for namespace is handled asynchronously #5188
Close RateLimiter instance #5155
Throw an error if the key was not specified for querying state #5145
Allow configuring region aware placement related settings #5100
DeleteBookieRack should remove the rack info from zookeeper #5084
Use "info" as the default root logger level #5079
Modify BatcherBuilder interface and it's subs to implement java.io.Serializable, otherwise java.io.NotSerializableException occurs when we use plusar-flink-connector #5068
Don't return DEFAULT_RACK if ZkBookieRackAffinityMapping can't resolve network location #5067
Reload zk cache asynchronously #5049
Add different cache flags to ConcurrentOpenLongPairRangeSet for size() and toString() #5040
Introduce number of threads in perf producer program #5036
Completing connector configuration #4999
Add checkstyle validation and fix style violations in the common module #4989
Trim deleted entries after recover cursor #4987
Expose getLastMessageId method in ConsumerImpl #4911
Add a documentation page for metrics reference #4910
Provide a convenient method for C++ client producer batch container #4885
Add schema admin api get schema info with schema version #4877
Return Message ID for send for cpp and cgo client #4811
Add the schema admin api #4800
Clarify how retention interacts with readers #4780
Get schema info with topic partition #4751
Remove failed stale producer from the connection #4741
Update logic for picking active consumer for failover subscription on non-partitioned topic #4604

2.4.1

2019-08-30

Fixes

Fixed wrong serialize of batchBuilder in ProducerConfigurationData #4620
Fixed SchemaInfo properties losing when registering schema using admin api #4617
Fixed wrong topic domain returned by get partitioned topic #4613
Fixed pulsar-dashboard wrong count number of bundles #4637
Fixed partitionIndex error in consumer for a single partitioned topic #4591
Fixed deadlock on get-status rest-api call in broker #4616
Fixed deadlock in subscribeAsync demo #4649
Fixed C++ client lookup error over HTTP in standalone #4625
Fixed NPE at managed-ledger when fetch reader internal-stats #4615
Fixed C++ client producer sendAsync() hang when no enough batched message #4657
Fixed issue when submitting NAR via file url #4577
Renamed C++ logger enum names to avoid conflicts with compiler macros #4664
Fixed leaking of pulsar-io-influxdb in distribution #4678
Fixed the default port for https and http in admin client #4623
Fixed wrong schema delete when checking compatibility #4669
Fixed docker/publish.sh doesn't work for pulsar-all and pulsar-standalone images #4705
Fixed integration-test failure when execute pip3 install pulsar_client #4754
Added log folder in pulsa-function-go back #4736
Fixed Pulsar SQL NPE when predicate pushdown for publish_time #4744
Fixed redelivered message logic of partition topic #4653
Fixed C++ log level names in Log4cxxLogger #4735
Fixed go function package not executable #4743
Added anonymous role to proxy configuration #4733
Added kubernetes namespace to function instance url #4701
Fixed go function not parse conf content first #4746
Made PulsarKafkaProducer thread safe #4745
Fixed messages not get acked if there is not sink topic #4815
Fixed bug when function package jar/py/go and runtime is not set #4814
Fixed topic loading in rest-api not time out in zooKeeperOperationTimeoutSeconds #4805
Fixed default retention policy miss upload to zk #4810
Added checking function implements correct interface #4844
Fixed retention size policy bug #4825
Catch throwable in interceptors of consumer and producer #4860
Fixed first position in managedLedger is ahead of the last #4853
Fixed concurrent access of uninitializedCursors in ManagedLedgerImpl.asyncOpenCursor #4837
Fixed schema not found handling in pulsar-sql #4890
Fixed requests not respect no_proxy env variable in dashboard #4867
Fixed broken replication msg to specific cluster #4930
Fixed dashboard peek parse message error #4918
Fixed StructSchema reader cache loading logic #4962
Fixed fd leakage in FunctionActioner.downloadFile #4970
Fixed python function str bytes convert in example custom_object_function.py #4946
Fixed NPE while cleaning up namespace node #4965
Fixed the getSchema logic in pulsar proxy #4975
Fixed warning by add default loader for latest pyyaml #4974
Fixed snappy compressor compile error in pulsar-client-cpp #4972
Reinitialize certain components for externally managed runtimes when moving functions #5007
Upgraded jackson-databind #5011
Fixed the problem of missing dependencies less in docker #5034
Fixed duplicated Prometheus TYPE in broker metrics #4183
Fixed pulsar sink and source state by init state earlier #5046

Enhancements

Re-factored Component implementation #4541
Provided a clock for generating publish timestamp for producers #4562
Reduced unnecessary track message calls #4595
Supported dynamic configure with escape char #4611
Added null check for function/source/sink configs #4627
Added delete dynamic config api #4614
Made broker replication mtls configuration dynamic #4609
Added authorization to function worker REST endpoints #4628
Improved and add authorization to function download and upload #4644
Allowed consumer retrieve the sequence id that the producer set #4645
Added perPartition parameter to partitioned-stats API #4639
Supported Pulsar schema for pulsar kafka client wrapper #4534
Supported delete and update event for JDBC Sink #4358
Cleaned up tests in the presto module #4683
Added allowAutoTopicCreation to broker.conf and related configuration #4694
Changed to use classloaders to load Java functions #4685
Removed fixed server type check in kerberos #4758
Changed type of publish_time to timestamp #4757
Added read-timeout to admin-request #4762
Added checking of deleted schema when adding schema #4731
Added strand to C++ client for exclusive control #4750
Added support to create partitioned topic with 1 partition #4764
Added getters and setters to PulsarService & BrokerService #4709
Added configure ack-timeout tick time #4760
Added options to rewrite namespace delimiter for pulsar sql #4749
Made Pulsar SQL supports pulsar's primitive schema #4728
Added basic authentication capabilities to Pulsar SQL #4779
Improved SchemaInfoProvider to fetch schema info asynchronously #4836
Supported KeyValue schema use AUTO_CONSUME as key/value schema #4839
Provided action type for insert in pulsar-io-jdbc #4862
Made partition as internal column in pulsar-sql #4888
Added option to disable authentication for proxy /metrics #4921

2.4.0

2019-06-30

PIPs

PIP-26: Added support for delayed message delivery #4062, note that messages are only delayed on shared subscriptions.
PIP-28: Pulsar Proxy Gateway Improvement #3915
PIP-29: One package for both pulsar client and pulsar admin #3662
PIP-30: Change authentication provider API to support mutual authentication for Kerberos Authentication #3677 #3821 #3997
PIP-32: Go Function API, Instance and LocalRun #3854 #4008 #4174
PIP-33: Replicated subscriptions to keep subscription state in-sync across multiple geographical regions #4299 #4340 #4354 #4396
PIP-34: Key_Shared subscription, Architecture for Key_Shared #4079 #4120 #4335 #4372 #4407 #4406 4462
PIP-36: Support set message size in broker.conf #4247

Fixes

Fix possible message loss using peer-cluster feature #3426
Fix pulsar standalone does not read zk port from conf/standalone.conf #3790
Fix some issues of ZkIsolatedBookieEnsemblePlacementPolicy #3917 #3918
Fix NPE when unload non-existent topic #3946
Fix race condition while deleting global topic #4173
Fix deadlock on skip messages #4411
Fix NPE when closing batch during a reconnection #4427
Fix race condition of read-timeout task in managed ledger #4437
Disable sticky read by default #4526
Fix race condition between timeout-task and add-call complete #4455

Enhancements

Optimize message replay for large backlog consumer #3732
Added support for websocket produce/consume command #3835
Added support for TTL config in broker.conf #3898
Reduce memory used in ClientCnx for pending lookups #4104
Reduce number of hashmap sections for ledger handles cache #4102
Added backlog and offloaded size in Prometheus stats #4150
Added support for configure the managed ledger cache eviction frequency #4066
Added support to avoid payload copy when inserting into managed ledger cache #4197
Added support to cache unack-messageId into OpenRangeSet #3819
Added support configure static PulsarByteBufAllocator to handle OOM errors #4196
Auto refresh new tls certs for jetty webserver #3645
Create non-persistent topic by pulsar-admin/rest api #3625
Consumer priority-level in Failover subscription #2954

Security

Added support for other algorithms in token auth #4528

Namespace Policies

Added support for tenant based bookie isolation #3933
Added support for secondary bookie isolation group at namespace #4458
Added support for secondary bookie-isolation-group #4261
Added support for replicator rate limit between clusters #4273
Disable backlog quota check by default #4320

Tiered Storage

Added support for Pulsar SQL to read data from tiered storage #4045

Pulsar Schema

Added schema versioning to support multi version messages produce and consume #3876 #3670 #4211 #4325 #4548
Added TRANSITIVE schema check strategies to support compatibility check over all existing schemas #4214
Added schema data validator #4360
Added support for delete schema when deleting a topic #3941
Added generic record builder #3690

Pulsar IO

Added IO connector for flume source and sink #3597
Added IO connector for redis sink #3700
Added IO connector for solr sink #3885
Hide kafka-connecter details for easy use debezium connector #3825
Added IO connector for debezium PostgreSQL source #3924
Enhancements for RabbitMQ source configuration #3937
Added IO connector for RabbitMQ sink #3967
Added IO connector for InfluxDB sink #4017

Pulsar functions

Added support for authentication #3735 #3874 #4198
Fix NPE when stats manager not initialized #3891
Added async state manipulation methods #3798
Fix fail to update functions in effectively-once mode #3993
Added labels to function statefulsets and services #4038
Added support for set key for message when using function publish #4005
Use negative acknowledge to instead ackTimeout #4103
Fix backward compatibility with 2.2 auth not working #4241

Java client

Added negative acks #3703
Added support for backoff strategy configuration #3848
Added support for configure TypedMessageBuilder through a Map conf object #4015
Added interceptor for negative ack send #3962
Added support for seek operate on reader #4031
Store key part of KeyValue schema into pulsar message keys #4117
Added interceptor for ack timeout #4300
Added support for snappy compression #4259
Added support for key based batcher #4435

Python client

Added negative acks #3816
Added support for snappy compression #4319

Go client

Added negative acks #3817
Added support for go schema #3904
Added support for snappy compression #4319
Added support for Key_Shared subscription #4465

C++ client

Added negative acks #3750
Fix ack timeout when subscribing to regex topic #3897
Added support for Key_Shared subscription #4366

Adaptors

In Kafka client wrapper, added some configurations #3753 #3797 #3843 #3887 #3991
In Apache Flink connector, allow to specify a custom Pulsar producer #3894 and client authentication #3949
In Apache Flink connector, added support for accept ClientConfigurationData, ProducerConfigurationData, ConsumerConfigurationData #4232
In Apache Storm connector, fix NPE while emitting next tuple #3991 and some add some enhancements #4280 #4239 #4238 #4236 #4495 #4494

For a complete list of issues fixed, see

https://github.com/apache/pulsar/milestone/20?closed=1

https://github.com/apache/pulsar/releases/tag/v2.4.0

2.3.2

2019-05-30

Fixes

Validate admin operation on topic with authoritative parameter #4270
fix bug with source local run #4278
fix cannot use size (type _Ctype_int) as type _Ctype_ulong #4212
Fix segfault in c++ producer #4219
AlwaysCompatible doesn't use AlwaysSchemaValidator in 2.3.1 component/schemaregistry #4181
Avoid potentially blocking method during topic ownership check #4190
[pulsar-broker]Fix: client-producer can't reconnect due to failed producer-future on cnx cache #4138
Removing # TYPE comment from topic metrics in Prometheus #4136
For functions metrics in prometheus also remove TYPE #4081
Fix: set receive queue size for sinks #4091
Fix: Exception when switch cluster from auth enabled to auth disabled #4069
Fix update cli source sink #4061
Fix connectors nested configs #4067
For functions metrics, avoid having HELP #4029
Fix Python functions state which is completely broken #4027
[issue #3975] Bugfix NPE on non durable consumer #3988
Fix: Function auth should ignore exception because it might be anonymous user #4185
[pulsar-function] fix worker-stats broken admin-api #4068
fix errors in sql doc #4030
Fix the swagger files generated by removing troublesome class #4024
[pulsar-function] fix broken backward compatibility with v1-namespace while registering function #4224
Revert dup consumer and related code #4142
[issue 4274][pulsar-io]Add double quotation marks for metrics with remote_cluster #4295

Enhancements

By default, auto configure the size of Bookie read/write cache #4297
Upgrade to BookKeeper 4.9.2 #4288
[pulsar-function] support bookie authentication from function-worker #4088
Optimizing performance for Pulsar function archive download #4082
allow users to update output topics for functions and sources #4092
improve data-generator source performance #4058
[client] Set actual topic name to partitioned consumer #4064
ack records in datagenerator print sink #4052
[security] Upgrade athenz libraries #4056
[python client] Handle subrecords in JsonSchema encoding #4023
[Issue 3987][pulsar-broker]Handle config is null when create tenant #4019
Add bookkeeper client version constraint #4013
Improve error handling for triggering function when there is a schema mismatch #3995
[pulsar-broker] add producer/consumer id in error-logging #3961

2.3.1

2019-04-12

Fixes

Fixed C++ batch acks tracker to evict message from sendList array. This was causing a slowdown in C++ consumers #3618
Allow publishing messages >5MB with batching (when they compress to <5MB) #3673 and #3718
Use at least 8 threads in Jetty thread pool. This fixed deadlocks in Jetty requests handling #3776
Fixed Reader.HasNext() in Go client #3764
Fixed increasing consumer permits after ack dedup operation. #3787
Set the dedup cursor as "inactive" after recovery #3612
Fix read batching message by pulsar reader #3830
Fix submit function with code specified via URL #3934
Fixed reader reading from a partition #3960
Fixed issue with Authorization header missing after client gets redirected #3869

Enhancements

Added producer.flush() on Python #3685
Introduced schema builder to define schema #3682
Support passing schema definition for JSON and AVRO schemas #3766
Exposing InitialPosition setting in Python consumer #3714

For a complete list of issues fixed, see

https://github.com/apache/pulsar/milestone/21?closed=1

https://github.com/apache/pulsar/releases/tag/v2.3.1

2.3.0

2019-02-20

General

Support for schema definitions in the Pulsar Python client library
PIP-25: Token based authentication #2888, #3067 and #3089
Updated to Apache BookKeeper 4.9.0
ZStandard compression codec #3159. Note that when a producer choose Zstd compression, a consumer will need to be at least at version 2.3.0 to be able to correctly receive the messages.
Support for Java 11 #3006
Added Schema.AUTO_PRODUCE type to allow to publish serialized data and validate it against the topic schema #2685
Added Schema.KeyValue to allow for schema to be validated on message keys as well as payloads. #2885
Support TLS authentication and authorization in standalone mode #3360
When creating namespace, use local cluster by default #3571
Tag BookKeeper ledgers created by Pulsar with topic/subscription names for info/debug purposes #3525
Enabled sticky reads in BookKeeper reads to increase IO efficiency with read-ahead #3569
Several optimization in Pulsar SQL Presto connector (#3128, #3135, #3139, #3144, #3143)
Configure Pulsar broker data cache automatically from JVM settings #3573
Reuse the SSL context objects #3550
Automatic schema update can be disabled through admin interface #2691
Support Dead-Letter-Queue from WebSocket proxy #2968
Pull-mode for WebSocket proxy #3058
Export Jetty stats to Prometheus #2804
Added stats for Pulsar proxy #2740
Allow subscribers to access subscription admin-api #2981
Make brokers read on closest Bookie in a multi-region deployment #3171

Fixes

Fixed deadlock in reusing ZooKeeper event thread #3591
In functions log topic appender, don't set producer name #3544
When cursor recovery encounters empty cursor ledger, fallback to latest snapshot #3487
Fixed C++ regex-consumer when using HTTP service URL #3407
Fix race condition: broker not scheduling read for active consumer #3411

Pulsar IO

Added Debezium connector for Change-Data-Capture into Pulsar #2791
Added MongoDB connector #3561
Added Elastic Search connector #2546
Added HBase sink #3368
Added Local files connector #2869
Report source/sink stats in Prometheus #3261
Allow filtering in Twitter Firehose connector #3298
Sources/Sinks can be launched using fat jars as well #3166

Pulsar Functions

Added Kubernetes runtime #1950
Secrets interface #2826
Cleanup subscriptions when deleting functions #3299
Add Windowfunction interface to functions api #3324
Support for accessing state in Python #2714
Support submitting Python functions as wheel file
Support submitting Python functions as Zip file with dependencies included #3321
Add minimum amount of resources to run setting for functions #3536
Fixed the behavior of Function start/stop #3477

Java client

Moved Pulsar v1 client API into separate artifact #3228.
Applications that are using the Pulsar v1 API, deprecated since 2.0 release, need to update the Maven dependency to use the pulsar-client-1x artifact instead of pulsar-client. Eg.
```
<dependency>
    <groupId>org.apache.pulsar</groupId>
    <artifactId>pulsar-client-1x</artifactId>
    <version>2.3.0</version>
</dependency>
```
Fixed shading issues with Javadoc bundled in client jars by separating the API in a different Maven module #3309
Improve Javadocs #3592
Support specifying multiple hosts in pulsar service url and web url #3249
Automatically discover when partitions on a topic are increased #3513
Added Client.getPartitionsForTopic() #2972 (Javadoc)
Added Consumer.pauseMessageListener() and Consumer.resumeMessageListener() #2961
Removed shading relocations for Circe-checksum and lz4 libraries, to ensure native libraries are correctly loaded when using shaded client lib. #2191

Python client

Fixed Message.properties() #3595

Go client

Added Producer.flush() to flush all outstanding messages #3469
Support Consumer.Seek() #3478
Added Message.Topic() #3346
Allow to specify SubscriptionInitPos option in ConsumerOptions #3588
Added TLS hostname verification #3580
Allow to link statically against libpulsar.a#3488
Expose Producer.LastSequenceID() and Message.SequenceID() #3416

C++ client

Enable batching by default when using sendAsync() #2949
Allow to specify schema info in Avro format #3354
Added Producer.flush() to flush all outstanding messages #3020
Added TLS hostname verification #2475
Allow to specify SubscriptionInitialPosition #3567
Added Message.getTopicName() #3326
Added Cosnsumer.receiveAsync() #3389
Build libpulsar.a with all required dependencies #3488
Removed Boost from Pulsar API headers #3374

Adaptors

Kafka client wrapper, added support for explicit partitioning and custom partitioner #3462
Support config auto.offset.reset to Pulsar KafkaConsumer #3273
In Apache Flink connector, added support for Batch Sink API (2979#, #3039 and #3046)
Added Java batch examples for Flink adaptor
Added Java streaming examples for Flink adaptor
Added Scala examples for Flink adaptor

For a complete list of issues fixed, see

https://github.com/apache/pulsar/milestone/18?closed=1

https://github.com/apache/pulsar/releases/tag/v2.3.0

2.2.1

2018-12-31

This release includes fixes for 2.2.0 release. In particular:

Fixed issue when proxy HTTP admin API requests Pulsar proxy #3022
Fixed Consumer.unsubscribe() in Python client library #3093
Fixed ZLib decompression in C++ client #2854
Fixed Pulsar functions context publish in Python #2844

For a complete list of issues fixed, see

https://github.com/apache/pulsar/milestone/19?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v2.2.1

2.2.0

2018-10-24

This is the first release of Pulsar as an Apache Top Level Project

This is a feature release, including several new features, improvements and fixes for issues reported for 2.1.1-incubating.

For a complete list of issues fixed, see

https://github.com/apache/pulsar/milestone/16?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v2.2.0

Apache incubator

2.1.1-incubating

2018-09-17

This release fixes issues reported for 2.1.0-incubating.

#2473 - Downgrading ZK to stable version 3.4.13
#2219 - Cpp client: add PatternMultiTopicsConsumerImpl to support regex subscribe
#2387 - Fixed race condition during expansion of concurrent open hash maps
#2348 - Fix NPE when splitting and unloading bundle
#2223 - fix bug in FunctionRuntimeManager involving not cleaning up old invalid assignments
#2367 - [compaction] make topic compaction works with partitioned topic
#2203 - Make sure schema is initialized before the topic is loaded

The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/17?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v2.1.1-incubating

2.1.0-incubating

2018-08-02

This is the seventh release of Apache Pulsar since entering the ASF incubator.

It is a feature release, including several new features and major improvements:

Pulsar IO: A connector framework for moving data in and out of Apache Pulsar leveraging Pulsar Functions runtime.
A list of Builtin Connectors:
Tiered Storage: An extension in Pulsar segment store to offload older segments into long term storage (e.g. HDFS, S3). S3 support is supported in 2.1 release.
Stateful function: Pulsar Functions is able to use State API for storing state within Pulsar.
Pulsar Go Client
Avro and Protobuf Schema support

The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/13?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating

2.0.1-incubating

2018-06-18

This release fixes issues reported for 2.0.0-rc1-incubating.

#1893 - Fixed issues with Python packages on PyPI
#1797 - Proxy doesn't strip the request URL for admin requests correctly
#1862 - Fix REST APIs provided by Pulsar proxy

The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/14?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v2.0.1-incubating

1.22.1-incubating

2018-06-18

This is the sixth release of Apache Pulsar since entering the ASF incubator.

This release addresses issues reported in 1.22.0-incubating version.

#1660 Deadlock while closing non persistent topic
#1591 Deadlock while closing non shared consumer
#1554 Handle invalid mark delete position at managed cursor
#1262 Broker should not start replicator for root partitioned topic
#1662 NPE when cursor failed to close empty subscription
#1370 Relocate service files for shading pulsar-client-admin module
#1265 Fixed lookup redirect logic on Proxyside
#1428 Handle Race condition in concurrent bundle split
#1817 Fixed mem leak when acknowledging while disconnected from broke
#1851 Fixing resource leak due to open file descriptors in SecurityUtility.

The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/15?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v1.22.1-incubating

2.0.0-rc1-incubating

2018-05-29

This is the fifth release of Apache Pulsar since entering the ASF incubator and the first time we increase major release number.

There are several new features and major improvements:

Pulsar functions: Lightweight compute framework
New type-safe Java API for producer/consumers
Schema registry — Enforce schema on topics
Topic compaction — Out of band compaction of messages to allow consumer to fetch a snapshot with last published message for each message key.
Upgraded to Apache BookKeeper 4.7.0
Performance improvements — Up to 3x throughput improvements compared to Pulsar-1.22 and 99.9 Pct publish latencies <10ms
Simplified terminology and admin tools
- Renamed "property" into "tenants"
- Short topic names: my-topic
- Topics independent of cluster names: my-tenant/my-namespace/my-topic

The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/12?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v2.0.0-rc1-incubating

1.22.0-incubating

2018-03-06

This is the fourth of Apache Pulsar since entering the ASF incubator.

Major changes in this release include:

Features

#896 PIP-7 Introduce Failure-domain and Anti-affinity-namespace group
#1031 Add optional key/value metadata to producers/consumers
#1129 Added end to end encryption in C++ client
#1151 Added REST handler to create a subscription on a topic
#1087 Add basic authentication plugin
#1200 Add pluggable authorization mechanism
#1208 Add hostname-verification at client tls connection
#950 Provided an DCOS Universe package for pulsar
#1046 Introduce config to skip non-recoverable data-ledger
#899 Add subscription auth mode by prefix
#1135 Added infinite time retention configuration option

Enhancements

#1094 Include BoringSSL native implementation for faster TLS
#1204 Reduce size of buffer used to assemble batches
#930 Perform async DNS resolution
#1124 Support Pulsar proxy from C++/Python client library
#1012 Made load shedding for load manager Dynamically configurable
#962 Raw Reader for Pulsar Topics
#941 Upgraded Jackson version
#1002, #1169, #1168 Making Pulsar Proxy more secure
#1029 Fix MessageRouter hash inconsistent on C++/Java client

Fixes

#1153 Fixed increase partitions on a partitioned topic
#1195 Ensure the checksum is not stripped after validation in the broker
#1203 Use duplicates when writing from ByteBuf pair to avoid multiple threads issues
#1210 Cancel keep-alive timer task after the proxy switch to TCP proxy
#1170 Upgrade BK version: BK-4.3.1.91-yahoo (fix: stats + DoubleByteBuf)
#875 Bug fixes for Websocket proxy

The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/11?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v1.22.0-incubating

1.21.0-incubating

2017-12-17

This is the third of Apache Pulsar since entering the ASF incubator.

Major changes in this release include:

#689 Upgrade to Netty 4.1
#846 Publish the shaded pulsar-client as the default dependency
#832 #833 #849 #852 Enhancements to Kafka API wrapper to have it work with Kafka's own benchmark tools
#836 Fix to C++ partitioned consumer client
#822 #826 Several fixes and improvements related to the namespace bundles
#848 Allow consumer to seek to message id from within Pulsar client
#903 PIP-8: Scale Pulsar beyond 1M topics
#824 Enable secure replication over TLS
#923 Upgrade to bk-4.3.1.83-yahoo to expose journalSyncData option
#807 Prevent message duplication when active consumer is changed

Complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/10?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v1.21.0-incubating

1.20.0-incubating

2017-08-08

This is the second of Apache Pulsar since entering the ASF incubator.

Major changes in this release include:

#620 #717 #718 Reader API support for C++, Python & Websocket Proxy
#634 Added Message dispatch throttling
#731 Added End to End Encryption
#732 Support for Event Time for messages
#751 Guaranteed Deduplication of Messages
#761 Kafka API wrapper for Pulsar client library

Complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/9?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v1.20.0-incubating

1.19.0-incubating

2017-08-08

This is the first of Apache Pulsar since entering the ASF incubator.

Major changes included in this release are:

#524 Moved APIs from com.yahoo.pulsar to org.apache.pulsar
#548 Added stateless Pulsar proxy
#538 Support for non-persistent topics
#587 Upgraded RocksDB to comply with ASF policy
#507 Instrumentation of ZooKeeper client to expose metrics
Various fixes for TLS auth in WebSocket proxy

Complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/8?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v1.19.0-incubating

Pre-Apache

1.18

2017-06-17

Main changes:

#325 Add Modular load manager documentation
#329 Add api to get list of partitioned topics
#296 Added spark streaming custom receiver for pulsar
#317 HTTP lookups for c++ client lib
#332 Fix: Modular load manager bug fixes
#352 Fix: Delete local-policies and invalidate cache when namespace is dele
#356 Fix: WebSocket TLS connection bug
#363 Use binary protocol lookup for connection between WebSocket proxy and broker
#375 Fix: Bug fixes on deadlock while topic loading failure
#376 Fix: Avoid incrementing unack-msg count for non-shared sub and not show it on stats
#329 Fix: Handle zkCache failures
#387 Pass client library version to broker and show on stats
#345 Add load shedding strategy
#393 Change default mark-delete rate limit from 10s to 1s
#392 Upgrade to netty-4.0.46
#366 NonDurable cursor for managed ledger
#371 Introduce topic reader in client API
#341 Add stats and monitoring for websocket proxy
#299 Add api to increase partitions of existing non-global partitioned-topic
#294 Add endpoint to fetch stats for Prometheus
#440 Enable PulsarAdmin to trust multiple certificates
#442 Fix: Remove broker weights for ModularLoadManager
#446 Fix: Recover cursor with correct readPosition and replay unackedMessages
#441 Set Block If queue full to false by default
#447 Fix: DoubleByteBuf to send large size messages in TLS mode
#443 Add topic termination option
#436 Added ZooKeeper instrumentation for enhanced stats
#448 WebSocket proxy should not make a consumer/producer when authorization is failed
#443 Add Docker images definition and instruction to deploy on Kubernetes
#474 Fix: message rate out with batches to count messages/s
#482 Allow client(producer/consumer) to check topic stats
#468 Pulsar Python client library
#386 Increment bookkeeper version to 4.3.1.69-yahoo

Full list of changes: https://github.com/yahoo/pulsar/milestone/7?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v1.18

1.17.5

2017-05-02

#343 Fix ModularLoadManager to select broker from current available-broker list
#384 Fix Send replay entries read callback from background thread, to avoid recursive stack calls
#390 Fix Shaded AsyncHttpClient in pulsar client
#374 Fix Remove Exceptionally Completed Topic Futures

https://github.com/apache/incubator-pulsar/releases/tag/v1.17.5

1.17.4

2017-04-25

#362 Fix add timeout on blocking ZookeeperCache get call
#375 Fix possible deal lock on topic loading if broker fails to get MLConfiguration from zk
#377 Fix zkCache error handling and zk-callback processing on separate dedicated thread

https://github.com/apache/incubator-pulsar/releases/tag/v1.17.4

1.17.3

2017-04-20

#367 Fix dispatcher correctly finds available consumer from list of shared-subscription consumers

https://github.com/apache/incubator-pulsar/releases/tag/v1.17.3

1.17.2

2017-04-06

#327 Create znode for dynamic configuration if not present
#336 Fix prevent creation of topic when bundle is disable
#338 Fix deserialize load report based on load-manager

https://github.com/apache/incubator-pulsar/releases/tag/v1.17.2

1.17.1

2017-03-30

#326 Fix memory leak while duplicating entry data from existing entry

https://github.com/apache/incubator-pulsar/releases/tag/v1.17.1

1.17

2017-03-30

Main changes:

#188 Pulsar Dashboard
#276 Broker persist individually deleted messages
#282 Support binary format to persist managed-ledger info in ZK
#292 Added REST and CLI tool to expose ManagedLedger metadata
#285 Add documentation in japanese
#178 Add Athenz authentication plugin
#186 Update Broker service configuration dynamically
#215 Fix Broker disconnects unsupported batch-consumer on batch-message topic
#165 Message dispatching on consumer priority-level
#303 Introduce new load manager implementation
#306 Add topic loading throttling at broker
#302 Update BK version to 4.3.1.60-yahoo to include: 64bit ledger-ids, fix: memory leak on read-only bookie and datasketches concurrency issue
#216 Binary proto api to get consumer stats
#225 Server lookup throttling
#182 Client lookup request throttling and server-error handling
#265 Fix client handling on http server error
#204 Fix discovery service redirection
#311 Fix netty package conflict at binary distribution
#221 Fixed race condition on client reconnection logic
#239 Fix replicator handling on closed cursor
#318 GC improvements: Recyclable entry and reduce collection on stats generation

Full list of changes: https://github.com/apache/incubator-pulsar/milestone/3?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v1.17

1.16.5

2017-03-10

#311 Exclude netty individual jars from binary distribution. This issue was causing binary distribution to have conflicting netty dependencies.

https://github.com/apache/incubator-pulsar/releases/tag/v1.16.5

1.16.4

2017-03-10

#265 Fix client closes http-connection on internal-server error
#283 Fix recycle keep alive command-object properly
#284 Reduce usage of collections in managed-ledger metrics-generation to reduce GC impact

https://github.com/apache/incubator-pulsar/releases/tag/v1.16.4

1.16.3

2017-03-01

#275 Fix for Infinite loop in PersistentReplicator.startProducer()

https://github.com/apache/incubator-pulsar/releases/tag/v1.16.3

1.16.2

2017-02-24

#250 : Disconnect consumers without closing dispatcher on cursor-reset

https://github.com/apache/incubator-pulsar/releases/tag/v1.16.2

1.16.1

2017-02-24

#221 Fixed race condition while creating client connection
#223 Fixed broker's direct memory usage count
#220 Fixed stuck replicator producer on backlog quota exception
#239 Fixed replicator stop reading on already closed cursor

https://github.com/apache/incubator-pulsar/releases/tag/v1.16.1

1.16

2017-02-02

Main changes:

#76 Async Zookeeper cache implementation
#105 Support topic lookup using pulsar binary protocol
#164 Fixed handling failure of unloading namespace bundle
#166 Support websocket proxy deployment without passing globalZK
#161 Fixed avoiding creation of duplicate replicator
#160 Add support uri encoding on broker admin rest api
#143 Include DataSketches metrics provider for bookie stats
#127 Updated BK-4.3.1.45/47-yahoo to include bookie/bookkeeper-client bug-fixes and DataSketch metrics provider
#124 Consumer-stats: Add blockedConsumer flag
#95 Consumer-stats: Add message redelivery rate
#123 Fixed Batch message replication
#106 Fixed Partitioned consumer should avoid blocking call to fill shared queue
#139 Support online consumer cursor reset
#187 Support custom advertised address in pulsar standalone

Full list of changes: https://github.com/yahoo/pulsar/milestone/2?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v1.16

1.15.7

2017-01-25

#174 Handling bundle unloading failure

https://github.com/apache/incubator-pulsar/releases/tag/v1.15.7

1.15.6

2017-01-20

#171 Fix: Consumer redelivery should not wipeout availablePermits

https://github.com/apache/incubator-pulsar/releases/tag/v1.15.6

1.15.5

2017-01-03

#159 Fix: Replicator-cleanup while closing replicator at broker.
#160 Fix: Http lookup for topic with special character

https://github.com/apache/incubator-pulsar/releases/tag/v1.15.5

1.15.4

2016-12-14

#146 Fix: Partitioned consumer can consume messages with receiverQueueSize 1.

https://github.com/apache/incubator-pulsar/releases/tag/v1.15.4

1.15.3

2016-12-13

#145 Fixed issue Partitioned-consumer aggregate messages without blocking internal listener thread

https://github.com/apache/incubator-pulsar/releases/tag/v1.15.3

1.15.2

2016-11-03

#102 Fixed issue with message dispatching while message-replay at broker

https://github.com/apache/incubator-pulsar/releases/tag/v1.15.2

1.15.1

2016-10-27

#89 Fixed issue with replication in a mixed environment with 1.14 and 1.15 brokers

https://github.com/apache/incubator-pulsar/releases/tag/v1.15.1

1.15

2016-10-18

#39 Updated BookKeeper version to 4.3.1.41-yahoo to include bookie storage improvements
#17 Fixed memory leak in stats generation buffer rollover
#27 Fixed issues with discovery service component when HTTPS is enabled
#43 Add end-to-end crc32c checksum verification on message header and payload, rather than just payload. Support for intel hardware instructions to speed up computation.
#26 Added ability to configure the address that the broker uses to advertise itself. Needed in cases where the public hostname/ip is different than the machine interface ip (eg: in AWS EC2 instances).
#38 Prevent message-replay of already acknowledged messages
#51 Per message unacknowledged redelivery. When ack-timeout is configured, only request redelivery of messages that effectively have the timeout expired, instead of all the messages dispatched to the consumer.
#48 Add unacknowledged messages threshold to stop delivery to consumer that are not acknowledging messages
#59 Added admin method to do a one-time messages time expiration for a given subscription (independently from the TTL configured at the namespace level)

Full list of changes: https://github.com/apache/incubator-pulsar/milestone/1?closed=1

https://github.com/apache/incubator-pulsar/releases/tag/v1.15

1.14

2016-08-31

First public release of Pulsar

https://github.com/apache/incubator-pulsar/releases/tag/v1.14

2.4.2​

2019-12-04​

Fixes​

Enhancements​

2.4.1​

2019-08-30​

Fixes​

Enhancements​

2.4.0​

2019-06-30​

PIPs​

Fixes​

Enhancements​

Security​

Namespace Policies​

Tiered Storage​

Pulsar Schema​

Pulsar IO​

Pulsar functions​

Java client​

Python client​

Go client​

C++ client​

Adaptors​

2.3.2​

2019-05-30​

Fixes​

Enhancements​

2.3.1​

2019-04-12​

Fixes​

Enhancements​

2.3.0​

2019-02-20​

General​

Fixes​

Pulsar IO​

Pulsar Functions​

Java client​

Python client​

Go client​

C++ client​

Adaptors​

2.2.1​

2018-12-31​

2.2.0​

2018-10-24​

Apache incubator​

2.1.1-incubating​

2018-09-17​

2.1.0-incubating​

2018-08-02​

2.0.1-incubating​

2018-06-18​

1.22.1-incubating​

2018-06-18​

2.0.0-rc1-incubating​

2018-05-29​

1.22.0-incubating​

2018-03-06​

Features​

Enhancements​

Fixes​

1.21.0-incubating​

2017-12-17​

1.20.0-incubating​

2017-08-08​

1.19.0-incubating​

2017-08-08​

Pre-Apache​

1.18​

2017-06-17​

1.17.5​

2017-05-02​

1.17.4​

2017-04-25​

1.17.3​

2017-04-20​

1.17.2​

2017-04-06​

2.4.2

2019-12-04

Fixes

Enhancements

2.4.1

2019-08-30

Fixes

Enhancements

2.4.0

2019-06-30

PIPs

Fixes

Enhancements

Security

Namespace Policies

Tiered Storage

Pulsar Schema

Pulsar IO

Pulsar functions

Java client

Python client

Go client

C++ client

Adaptors

2.3.2

2019-05-30

Fixes

Enhancements

2.3.1

2019-04-12

Fixes

Enhancements

2.3.0

2019-02-20

General

Fixes

Pulsar IO

Pulsar Functions

Java client

Python client

Go client

C++ client

Adaptors

2.2.1

2018-12-31

2.2.0

2018-10-24

Apache incubator

2.1.1-incubating

2018-09-17

2.1.0-incubating

2018-08-02

2.0.1-incubating

2018-06-18

1.22.1-incubating

2018-06-18

2.0.0-rc1-incubating

2018-05-29

1.22.0-incubating

2018-03-06

Features

Enhancements

Fixes

1.21.0-incubating

2017-12-17

1.20.0-incubating

2017-08-08

1.19.0-incubating

2017-08-08

Pre-Apache

1.18

2017-06-17

1.17.5

2017-05-02

1.17.4

2017-04-25

1.17.3

2017-04-20

1.17.2

2017-04-06