Airbnb’s Mussel V2: The Unified Engine for Streaming and Bulk Data

Airbnb’s Mussel V2 represents a major step forward in the company’s data infrastructure. Designed as a unified key-value storage system, it seamlessly integrates real-time streaming ingestion with bulk batch processing, addressing the growing challenge of managing heterogeneous data pipelines at scale.

Why Airbnb Needed a Storage Redesign

At Airbnb’s scale, data arrives from multiple systems — transactional services, analytics pipelines, and machine learning models. The first version of Mussel served as a distributed KV store optimized for read-heavy workloads. However, as event streaming (via Kafka) and bulk imports grew in importance, Airbnb engineers faced a mismatch between streaming and batch ingestion capabilities.

Mussel V2 was built to unify these two worlds — providing consistent performance and storage semantics regardless of ingestion mode.

Core Innovations in Mussel V2

1. Dual Ingestion Architecture

Mussel V2 supports both streaming and bulk ingestion through a shared commit protocol. The streaming path uses Kafka topics for continuous updates, while the batch path leverages distributed compaction for large-scale imports. This eliminates the need for separate systems for hot and cold data.

2. Unified Data Model

Rather than maintaining two versions of the same dataset for different ingestion modes, Mussel V2 employs a unified record schema. Each key maintains a version history and timestamped metadata, ensuring consistency even during concurrent writes from different sources.

3. Optimized Write Path and Compaction

The system integrates adaptive compaction — inspired by LSM-tree techniques — to reduce write amplification. Hot partitions receive more frequent compaction, while cold data segments are compacted lazily, reducing system overhead.

Performance and Scalability

• Write throughput: Mussel V2 improves write performance by up to 3× under mixed ingestion workloads.
• Latency reduction: End-to-end write latency dropped by 45% compared to the original Mussel.
• Horizontal scaling: Storage nodes can dynamically rebalance partitions without full data reloads.

These optimizations enable Airbnb to support both analytical workloads (feature stores, recommendations) and operational services (inventory, pricing) from a unified infrastructure layer.

Engineering Challenges

One of the biggest hurdles was maintaining ordering guarantees between streaming and batch updates. The team developed a hybrid timestamp reconciliation algorithm that merges event timelines deterministically across ingestion modes.

Another challenge was ensuring backward compatibility for legacy services that still rely on Mussel V1 APIs. A compatibility shim allows gradual migration without service downtime.

Integration Across Airbnb’s Ecosystem

Mussel V2 now underpins several mission-critical systems, including feature stores for personalization and near-real-time dashboards. It also integrates with Airbnb’s internal orchestration frameworks and metadata services, enabling easier data lineage tracking and observability.

Looking Ahead

Future plans for Mussel V2 include:

• Support for tiered storage using object stores like S3.
• Extended query APIs for real-time analytics.
• Improved self-healing and automated load balancing.

Conclusion

With Mussel V2, Airbnb has redefined its approach to large-scale data management. By merging streaming and bulk ingestion into one coherent system, the company gains performance, operational simplicity, and a clear path toward fully unified data infrastructure.