Since Kafka does not support delayed messages, and our company’s technology stack uses Kafka as the message middleware, the business side wants to use RocketMQ to meet the delayed message scenario. However, introducing another message middleware just for delayed messaging would increase operational and maintenance costs. Against this background, we hope to extend the Kafka client to support delayed messages.
This article will introduce four delayed message implementation schemes, analyzing their advantages and disadvantages.
Scheme One: Time Wheel Algorithm
Each producer holds a time wheel delayed message queue, with messages stored in memory.
- slot = (current timestamp - time wheel start time) % total slots;
- round = (current timestamp - time wheel start time) / total slots;
- Delayed message chain: sorted by round.
Disadvantage analysis:
- Consumes significant memory resources, only suitable for short delay times (e.g., within one minute);
- Prone to message loss.
Scheme Two: Single-Round Time Wheel Algorithm + File Storage (Improved Version of Scheme One)
Using a single-round time wheel algorithm, with each slot pointing to a file.
Disadvantage analysis:
- If the service is containerized and rebuilt, the time wheel file may be lost, compromising message consistency.
Scheme Three: Multi-Level Partitioning + Automatic Downgrading
Divide messages into multiple partitions based on remaining delay time (expected send time - current time), downgrading messages to a specific partition until reaching the actual topic.
Disadvantage analysis:
- Requires multiple sends and subscriptions; for example, a 2-hour delayed message would need at least 8 subscriptions and 9 sends to reach the target topic;
- Due to varying delay times within the same level, ensuring accurate delay times might lead to a single message requiring hundreds of sends and subscriptions.
For instance, if the delay times in a queue are 50, 40, 36, and 56 minutes, each message must be consumed and rewritten to the same level partition to avoid affecting subsequent delayed messages. In the worst case, a high-delay message might require over 100 sends and subscriptions.
Scheme Four: Multi-Level Delay, Without Supporting Arbitrary Time Precision (Improved Version of Scheme Three)
Referencing RocketMQ’s delayed message design, which does not support arbitrary time precision, but instead supports specific delay levels. Divide delay levels into 1s, 5s, 10s, 30s, 1m, 2m, 3m, 4m, 5m, 6m, 7m, 8m, 9m, 10m, 20m, 30m, 1h, and 2h, with 18 levels in total. Create a delayed topic with 18 partitions, each corresponding to a different delay level.
Example:
- When sending a 5-second delayed message, send it to the second partition of the order-topic.delay topic;
- When sending a 1-minute delayed message, send it to the fifth partition of the order-topic.delay topic;
- When sending a 1-hour delayed message, send it to the 17th partition of the order-topic.delay topic;
Advantages:
- Ensures that messages within each partition are in chronological order, only requiring sequential consumption of each partition and forwarding messages that have reached their send time to the actual topic;
- If a message has not reached its send time, there is no need to commit the offset, as messages with later offsets in the same partition are also guaranteed to be undelivered.
We ultimately adopted Scheme Four. In our implementation, we started a KafkaConsumer for each process, using regular expressions to subscribe to topics ending with ‘.delay’, reducing thread resource consumption. When sending messages to the delayed topic, we used the delay level as the message key and stored the original message key in the message header. When sending to the actual topic, we retrieved the real key and real topic from the delayed message header.