Mainstream Distributed ID Solutions

1. Introduction

In high-throughput systems, such as an e-commerce platform during a promotion, it is critical to quickly assign every order a globally unique, conflict-free ID.

If you rely only on database auto-increment IDs, the database becomes the bottleneck as soon as throughput grows; with multiple instances, there is no guarantee that IDs generated by different nodes are globally unique and conflict-free; completely random, out-of-order IDs can also cause frequent page splits in database indexes; and IDs may expose sensitive system information, such as business volume.

These are the problems distributed IDs need to solve: global uniqueness, high availability and performance, trend-increasing order, and security.

As a result, many approaches have emerged: the bit-partitioning Snowflake algorithm, the segment mode that generates IDs in batches, and purely local UUIDs, among others.

2. ID Formats: Four Main Types and Their Characteristics

There are roughly four ID formats:

Time-ordered: Snowflake, UUID v1/v6/v7 They place a timestamp in the high-order bits and append random numbers, sequence numbers, and node IDs, ensuring chronological ordering. Pros: time-ordered, generated locally Cons: clock rollback, time leakage

Random: UUID v4 It does not depend on time or the machine; it is completely random. Pros: random, does not expose information, generated locally Cons: unordered, unfriendly to indexes

Segment distribution: DB, Leaf Segment, TinyID, Redis A batch of numbers (a segment) is fetched into memory, and IDs are issued with local auto-increment. Pros: easy to scale, trend-increasing order, smoother and more stable with dual-buffer optimization Cons: leaks business volume, relies on DB or Redis to maintain allocation, and horizontal scaling can only guarantee trend-increasing order

Ring-buffer prefetch: uID-generator Drawing on segment mode and Snowflake, it uses Snowflake's bit-allocation idea to build ordered IDs and fills a ring array as a cache pool. Its timestamp is set only at initialization; afterwards, each refill increments the timestamp and takes all the IDs for that instant. Pros: ordered, generated locally, can exceed the one-second sequence-number cap for high concurrency Cons: the timestamp in the ID no longer represents real time, but it can borrow future time in advance

3. Implementations: Seven Distributed ID Generation Schemes

3.1 Snowflake: Twitter's Open-Source Bit-Partitioned Time-Ordered ID Scheme

Snowflake is a distributed ID generation algorithm open-sourced by Twitter. Its core is to partition the bit space so each field identifies one dimension.

Bit allocation and structure

The 41-bit timestamp has millisecond granularity and can hold $2^{41}ms$, roughly 69 years. Putting the timestamp in the high-order bits ensures that IDs generated across the system increase overall.

The two 5-bit identifier fields can provide $2^{10}=1024$ machine IDs, and the 12-bit auto-increment sequence can represent $2^{12}=4096$ IDs.

So theoretically, a single machine using Snowflake can generate IDs at a rate of $2^{12}\times1000=409.6w/s$.

Issues

It depends heavily on the clock. If clock rollback occurs, IDs may become abnormal—for example, duplicated or out of order.

Clock rollback

1. Block directly until lastTimestamp.
2. Maintain an offset that grows during rollback, and actually use logicalTimestamp = currentTimestamp + offset.
3. Borrow high-order bits as a rollback tag.

3.2 Database Generation: ID Scheme Based on Database Auto-Increment

This approach uses database auto-increment to generate increasing IDs by assigning different machines different initial values and the same step size equal to the number of machines.

Drawbacks

It depends heavily on the DB and is not easy to scale, because the step size is based on the number of machines. A single-machine DB limits the ID issuance ceiling; with multiple machines, only trend-increasing order can be guaranteed, and maintenance is harder. The DB bears heavy load because every ID fetch requires a read and a write.

3.3 UUID Series: Universally Unique Identifier

UUID (Universally Unique Identifier), also called GUID, consists of 32 hexadecimal digits in the form xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx, where M and N represent the version and variant respectively.

UUID v1

60-bit timestamp + 14-bit clock sequence + 48-bit node (commonly MAC or pseudo-random) v1's 60-bit timestamp (in 100ns units, from 1582-10-15) is split into time_low(32) + time_mID(16) + time_high(12).

UUID v4

Except for the version/variant bits, it is almost entirely random (122 bits of entropy). It does not expose time or host information. JDK's UUID.randomUUID() is v4.

UUID v6

A rearranged version of v1, ordered by the high-order time bits, making it more suitable for database inserts. 60-bit timestamp + 14-bit clock sequence + 48-bit node (commonly MAC or pseudo-random) The 60-bit timestamp is split into time_high(32)+time_mID(16)+time_low(12).

UUID v7

48-bit millisecond timestamp + 12-bit randa + variant + randb (remaining random/monotonic fields)

UUID v7 adopts the concept of ULID.

ULID: Universally Unique Lexicographically Sortable Identifier

48 bits │ 80 bits 48-bit millisecond timestamp 80-bit secure random number

Notably, ULID uses Base32 encoding (digits + letters, excluding I, L, O, U), unlike UUID's Base16 encoding. It ends up as a 26-character string, e.g. 01FZ4Q3YSJH9X8HRXW4E7JKX4N

3.4 Meituan Leaf: Highly Available Distributed ID Generation Service

Meituan Tech: Leaf – Meituan-Dianping Distributed ID Generation System(https://tech.meituan.com/2017/04/21/mt-leaf.html?utm_source=chatgpt.com)

Leaf Segment

This optimizes the database approach: the original scheme requests the database once per ID, while the improved version uses a proxy server to fetch IDs in batches. Each time, it gets a segment of numbers, and after that segment is used up, it fetches another one.

biz_tag distinguishes the business; each fetch obtains step IDs, reducing database read/write frequency from 1 to 1/step.

Pros

1. IDs are trend-increasing.
2. It can scale linearly.

Cons

1. IDs are auto-incrementing and not random enough, which may leak information.
2. When the current segment is exhausted and a new one is fetched, there is a large spike.

Dual-buffer optimization

The ID consumption process is not smooth in this scheme; when a segment is exhausted and a new segment is fetched, there is a big fluctuation. To make consumption non-blocking, you can asynchronously fetch the next segment when the current one reaches a threshold, instead of waiting until it is exhausted and then blocking while fetching a new one.

In other words, use two segment buffers. When the currently active segment reaches a threshold, asynchronously fetch the next one; when the active segment runs out, switch directly to the other buffer and keep issuing IDs, making issuance stable.

Leaf Snowflake

Leaf Segment can stably produce trend-increasing IDs, but it leaks data volume, which led to Leaf Snowflake.

It still follows Snowflake's bit-partitioning design.

Bit allocation

1 | 41 | 10 | 12 1-bit unused 41-bit millisecond timestamp 10-bit workerID 12-bit sequence number, 0–4095 per millisecond

Clock rollback

ZooKeeper is used for time consistency and health detection.

/leaf_forever/{self}: persistent node that stores the instance's last reported system time (ms), used for startup and runtime comparison to detect whether a large clock rollback has occurred. /leaf_temporary/{self}: temporary node that periodically during runtime (e.g., every 3s) reports local time and pulls the times of all online nodes to compute the mean or median.

3.5 Didi TinyID: Segment Mode Supporting Multiple Master DBs

Tinyid原理介绍

ID Generator id生成器 分布式id生成系统，简单易用、高性能、高可用的id生成系统. Contribute to didi/tinyid development by creating an account on GitHub.

GitHub

It is similar in principle to Leaf Segment and also uses a dual-buffer design, but it additionally supports multiple master DBs. Even if the same segment is fetched, the IDs do not overlap: ID % delta == remainder.

biz_type: business type max_ID: maximum ID (multi-master support) step: step size, i.e., the size of one segment delta: increment per ID, used to support multiple DBs remainder: remainder, used to support multiple DBs

Example

With three master DBs, A, B, and C, each can fetch the same segment (3000, 4000], but each one first aligns to its own first: first = start + ((end - start % delta + delta) % delta) That is, each DB's segment can only supply 1/delta of the IDs.

It can also generate only odd IDs, for example with delta = 2, remainder = 1.

3.6 Baidu uID-generator: Pre-fillable Snowflake Algorithm Based on Dual-RingBuffer Optimization

Bit allocation

1 | 28 | 22 | 13 1-bit unused 28-bit second-level timestamp 22-bit workerID 13-bit sequence number, 0–8191 per second

DefaultUIDGenerator

This is the standard Snowflake.

synchronized
获取当前时间
当前时间小于上次时间
    抛异常，时钟回拨
当前时间等于上次时间
    序列号自增，同时取模最大序列号（也就是 & 8191）
    序列号等于 0（也就是超过当前秒所能供应的上限）
        阻塞至下一秒并修改当前时间
当前时间是最新的
    序列号置为 0
上次时间置为当前时间
构造 ID

CachedUIDGenerator

It uses dual RingBuffers: UID-RingBuffer stores UIDs, and Flag-RingBuffer stores UID states (CANTAKE, CANPUT).

RingBuffer is a ring array with a producer pointer (tail) and a consumer pointer (cursor). Its capacity is the maximum sequence number per second, 8192.

Getting an ID takes it directly from the RingBuffer.

Filling IDs

• At startup: fill the RingBuffer with 8192 IDs
• When getUID() is called, check the capacity; if the remaining IDs fall below a threshold, e.g. 50%, refill it
• Refill on a timer

To generate IDs, the first ID is constructed from the passed-in currentSecond, and then the ID list is filled to the maximum sequence count for that second.

Filling IDs obtains all IDs from the second after lastSecond and fills them one by one until the RingBuffer is full.

Drawbacks

In this scheme, lastSecond may differ from real time. Except at initialization, when it is set to real time, it is incremented in all other cases to obtain the IDs for that moment. Therefore, the timestamp in the IDs cannot represent real time in most cases.

Slow consumption is likely to keep falling behind real time. Fast consumption is likely to stay far ahead of real time (future time can be borrowed in advance, because filling directly increments lastSecond).

3.7 JD In-Memory Segment: Segment Mode Based on Redis + In-Memory Auto-Increment

Following JD's short-link design, short-code ID distribution works like Leaf Segment: it gets a segment from the Redis cache and generates IDs by in-memory auto-increment.

It can leverage the dual-buffer optimization and Redis's own INCRBY atomic increment.