Apache Pulsar: Distributed Pub-Sub Messaging System

Apache Pulsar is an open-source distributed pub-sub messaging system originally created at Yahoo and part of the Apache Software Foundation.

Pulsar is a multi-tenant, high-performance solution for server-to-server messaging .

Pulsar’s key features include [4] :

Architecture overview

At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can replicate data amongst themselves [4].

The diagram below provides an illustration of a Pulsar cluster:

Pulsar System Architecture

Pulsar Comparison with Apache Kafka

The table below lists the similarities and differences between Apache Pulsar and Apache Kafka [5]:

KafkaPulsar
ConceptsProducer-topic-consumer group-consumerProducer-topic-subscription-consumer
ConsumptionMore focused on streaming, exclusive messaging on partitions. No shared consumption.Unified messaging model and API.

  • Streaming via exclusive, failover subscription
  • Queuing via shared subscription
AckingSimple offset management

  • Prior to Kafka 0.8, offsets are stored in ZooKeeper
  • After Kafka 0.8, offsets are stored on offset topics
Unified messaging model and API.

  • Streaming via exclusive, failover subscription
  • Queuing via shared subscription
RetentionMessages are deleted based on retention. If a consumer doesn’t read messages before retention period, it will lose data.Messages are only deleted after all subscriptions consumed them. No data loss even the consumers of a subscription are down for a long time.

Messages are allowed to keep for a configured retention period time even after all subscriptions consume them.

TTLNo TTL supportSupports message TTL

Conclusion

Apache Pulsar is an effort undergoing incubation at The Apache Software Foundation (ASF) [3] sponsored by the Apache Incubator PMC. It seems that it will be a competitive alternative to Apache Kafka due to its unique features.

Resources:

[1] https://pulsar.apache.org/

[2] https://developer.yahoo.com/open-source/

[3] https://apache.org/

[4] https://pulsar.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/

[5] https://streaml.io/blog/pulsar-streaming-queuing/

kamaci

Leave a Reply

Your email address will not be published. Required fields are marked *