Skip to main content

End-to-End Encryption

Applications can use Pulsar end-to-end encryption (E2EE) to encrypt messages on the producer side and decrypt messages on the consumer side. You can use the public and private key pair that the application configures to perform encryption and decryption. Only the consumers with a valid key can decrypt the encrypted messages.

How end-to-end encryption works in Pulsar​

Pulsar uses a dynamically generated symmetric session key to encrypt messages (data). You can use the application-provided ECDSA (Elliptic Curve Digital Signature Algorithm) or RSA (Rivest–Shamir–Adleman) key pair to encrypt the session key (data key), so you do not have to share the secret with everyone.

The following figure illustrates how Pulsar encrypts messages on the producer side and decrypts messages on the consumer side.

End-to-end encryption in Pulsar

The workflow of end-to-end encryption in Pulsar is as follows.

  1. The producer generates a session key regularly (every 4 hours or after publishing a certain number of messages) to encrypt the message payload using a symmetric algorithm, such as AES, and fetches the asymmetric public key every 4 hours. The ciphertext is packed as the message body.
  2. The producer uses the consumer’s public key to encrypt the session key using an asymmetric algorithm, such as RSA, and adds an alias with the encrypted secret to the message header.
  3. The consumer reads the message header and decrypts the session key using its private key.
  4. The consumer uses the decrypted session key to decrypt the message payload.
note
  • The consumer's public key is shared with the producer, but only the consumer has the access to the private key.
  • Pulsar does not store the encryption key anywhere in the Pulsar service. If you lose or delete the private key, your message is irretrievably lost and unrecoverable.

Pulsar isolates the key management and only provides interfaces (CryptoKeyReader) to access public keys. For production systems, it's highly recommended to extend/implement CryptoKeyReader with cloud key management (KMS or CKM) or PKI (Public Key Infrastructure, such as freeIPA).

If the produced messages are consumed across application boundaries, you need to ensure that consumers in other applications have access to one of the private keys that can decrypt the messages. You can do this in two ways:

  • Access the public key that the consumer application provides and add it to the producer's keys.
  • Grant access to one of the private keys from the pairs that the producer uses.

Get started​

To enable end-to-end encryption in Puslar, complete the following steps.

Prerequisites​

  • Pulsar Java/Python/C++/Node.js client 2.7.1 or later versions.
  • Pulsar Go client 0.6.0 or later versions.

Step 1: Configure end-to-end encryption​

  1. Create both public and private key pairs.

    openssl ecparam -name secp521r1 -genkey -param_enc explicit -out test_ecdsa_privkey.pem
    openssl ec -in test_ecdsa_privkey.pem -pubout -outform pem -out test_ecdsa_pubkey.pem

  2. Configure a CryptoKeyReader on producers, consumers or readers.

    PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
    String topic = "persistent://my-tenant/my-ns/my-topic";
    // RawFileKeyReader is just an example implementation that's not provided by Pulsar
    CryptoKeyReader keyReader = new RawFileKeyReader("test_ecdsa_pubkey.pem", "test_ecdsa_privkey.pem");

    Producer<byte[]> producer = pulsarClient.newProducer()
         .topic(topic)
         .cryptoKeyReader(keyReader)
         .addEncryptionKey("myappkey")
         .create();

    Consumer<byte[]> consumer = pulsarClient.newConsumer()
         .topic(topic)
         .subscriptionName("my-subscriber-name")
         .cryptoKeyReader(keyReader)
         .subscribe();

    Reader<byte[]> reader = pulsarClient.newReader()
         .topic(topic)
         .startMessageId(MessageId.earliest)
         .cryptoKeyReader(keyReader)
         .create();

  3. Optional: customize the CryptoKeyReader implementation.

    class RawFileKeyReader implements CryptoKeyReader {

     String publicKeyFile = "";
     String privateKeyFile = "";

     RawFileKeyReader(String pubKeyFile, String privKeyFile) {
         publicKeyFile = pubKeyFile;
         privateKeyFile = privKeyFile;
     }

     @Override
     public EncryptionKeyInfo getPublicKey(String keyName, Map<String, String> keyMeta) {
         EncryptionKeyInfo keyInfo = new EncryptionKeyInfo();
         try {
             keyInfo.setKey(Files.readAllBytes(Paths.get(publicKeyFile)));
         } catch (IOException e) {
             System.out.println("ERROR: Failed to read public key from file " + publicKeyFile);
             e.printStackTrace();
         }
         return keyInfo;
     }

     @Override
     public EncryptionKeyInfo getPrivateKey(String keyName, Map<String, String> keyMeta) {
         EncryptionKeyInfo keyInfo = new EncryptionKeyInfo();
         try {
             keyInfo.setKey(Files.readAllBytes(Paths.get(privateKeyFile)));
         } catch (IOException e) {
             System.out.println("ERROR: Failed to read private key from file " + privateKeyFile);
             e.printStackTrace();
         }
         return keyInfo;
     }
    }

Step 2: Encrypt a message with multiple keys​

note

This is only available for Java clients.

You can encrypt a message with more than one key. Producers add all such keys to the config and consumers can decrypt the message as long as they have access to at least one of the keys. Any one of the keys used for encrypting the message is sufficient to decrypt the message.

For example, encrypt the messages using 2 keys (myapp.messagekey1 and myapp.messagekey2):

PulsarClient.newProducer().addEncryptionKey("myapp.messagekey1").addEncryptionKey("myapp.messagekey2");

Troubleshoot​

  • Producer/Consumer loses access to the key
    • Producer action fails to indicate the cause of the failure. Application has the option to proceed with sending unencrypted messages in such cases. Call PulsarClient.newProducer().cryptoFailureAction(ProducerCryptoFailureAction) to control the producer behavior. The default behavior is to fail the request.
    • If consumption fails due to decryption failure or missing keys in the consumer, the application has the option to consume the encrypted message or discard it. Call PulsarClient.newConsumer().cryptoFailureAction(ConsumerCryptoFailureAction) to control the consumer behavior. The default behavior is to fail the request. Application is never able to decrypt the messages if the private key is permanently lost.
  • Batch messaging
    • If decryption fails and the message contains batch messages, client is not able to retrieve individual messages in the batch, hence message consumption fails even if cryptoFailureAction() is set to ConsumerCryptoFailureAction.CONSUME.
  • If decryption fails, the message consumption stops and the application notices backlog growth in addition to decryption failure messages in the client log. If the application does not have access to the private key to decrypt the message, the only option is to skip or discard backlogged messages.
Was this helpful?