← Back

Implementing a Distributed Lock on S3

5 min read

The Problem: Only One Worker at a Time

When you scale an application out across many independent compute nodes — microservices, background workers, cron-style jobs — one requirement shows up constantly: only one worker may perform a specific critical task at any given time.

Typical examples:

  1. Multiple worker nodes fetching unprocessed orders, calculating totals, and generating invoices.
  2. Multiple consumers reading from a single partition of a stream.
  3. Multiple processes performing a read-modify-write cycle on a shared storage file.

If two workers execute any of these simultaneously without coordination, you get race conditions: a customer gets double-charged, a job runs twice, or two workers overwrite each other's progress in storage — the classic lost update.

What is a Distributed Lock?

A distributed lock provides mutual exclusion across uncoordinated nodes on a network. Nodes compete for a single lock, and all of them agree on one rule: whichever node holds the lock is the designated leader for that task. Everyone else stands down until the lock changes hands. (For a longer primer on why single-machine mutexes don't work here, see the distributed lock note in Building a Database on S3.)

Historically, building one on S3 meant leaning on external coordination services like ZooKeeper or DynamoDB, because S3 had no atomic Compare-and-Swap primitives. That changed in 2024 when AWS added conditional writes to S3: a PutObject with the If-None-Match: "*" header creates the object only if no object with that key exists yet — otherwise S3 returns 412 Precondition Failed. One HTTP header turned S3 itself into a lock service. (The full timeline of S3's consistency and CAS primitives is covered in Building a Database on S3.)

The Algorithm: Epochs and Conditional Writes

The scheme below (from Gunnar Morling's leader election design) structures leadership into sequential epochs:

  1. Epoch files: lock files are named sequentially with zero-padded integers — lock-0000000001.json, lock-0000000002.json, and so on.
  2. Lock acquisition:
    • A node lists existing lock objects to find the highest epoch number.
    • It attempts to create the next epoch's file with If-None-Match: "*".
    • If the write succeeds, the node is the leader for that epoch.
    • If S3 returns 412, another node created it first — the attempt fails.
  3. Lock release and expiration: the leader can gracefully step down by marking its epoch file "expired": true, and each payload carries a validity_ms lease so crashed leaders don't hold the lock forever.

In pseudo-code:

text
acquire_lock():
  latest  = list lock files, take highest epoch number
  next    = latest + 1
  key     = "lock-" + pad(next, 10) + ".json"
 
  result  = S3.put(key, {leader, validity_ms},
                  if_none_match: "*")    # create-only
 
  if result is OK:   return LEADER of epoch <next>
  if result is 412:  another node won; return NOT_LEADER
 
release_lock(epoch):
  S3.put("lock-" + pad(epoch, 10) + ".json",
         {expired: true})
 
still_valid(epoch):
  (now - lock.created_at) < validity_ms AND not expired

TypeScript Implementation

typescript
import {
  S3Client,
  PutObjectCommand,
  ListObjectsV2Command,
  S3ServiceException,
} from "@aws-sdk/client-s3";
 
interface LockPayload {
  leaderId: string;
  validityMs: number;
  expired: boolean;
}
 
export class S3DistributedLock {
  private s3: S3Client;
  private bucket: string;
  private lockPrefix: string;
  private leaderId: string;
 
  constructor(
    bucket: string,
    lockPrefix = "locks/lock-",
    leaderId = process.env.HOSTNAME || "node-1",
  ) {
    this.s3 = new S3Client({});
    this.bucket = bucket;
    this.lockPrefix = lockPrefix;
    this.leaderId = leaderId;
  }
 
  /** Finds the latest epoch number from existing S3 lock files. */
  private async getLatestEpoch(): Promise<number> {
    const response = await this.s3.send(
      new ListObjectsV2Command({
        Bucket: this.bucket,
        Prefix: this.lockPrefix,
      }),
    );
 
    if (!response.Contents || response.Contents.length === 0) {
      return 0;
    }
 
    let maxEpoch = 0;
    for (const obj of response.Contents) {
      if (!obj.Key) continue;
      // Extract numeric epoch from "locks/lock-0000000001.json"
      const match = obj.Key.match(/lock-(\d+)\.json$/);
      if (match) {
        const epoch = parseInt(match[1], 10);
        if (epoch > maxEpoch) maxEpoch = epoch;
      }
    }
    return maxEpoch;
  }
 
  /**
   * Attempts to acquire leadership for the next epoch using
   * an S3 conditional write (If-None-Match: "*").
   */
  async tryAcquireLock(
    validityMs = 60000,
  ): Promise<{ acquired: boolean; epoch: number }> {
    const latestEpoch = await this.getLatestEpoch();
    const nextEpoch = latestEpoch + 1;
    const lockKey =
      `${this.lockPrefix}${String(nextEpoch).padStart(10, "0")}.json`;
 
    const payload: LockPayload = {
      leaderId: this.leaderId,
      validityMs,
      expired: false,
    };
 
    try {
      // Create-only: succeeds iff no object exists at this key
      await this.s3.send(
        new PutObjectCommand({
          Bucket: this.bucket,
          Key: lockKey,
          Body: JSON.stringify(payload),
          ContentType: "application/json",
          IfNoneMatch: "*",
        }),
      );
 
      // Success means this worker created the file and won
      return { acquired: true, epoch: nextEpoch };
    } catch (err) {
      if (
        err instanceof S3ServiceException &&
        (err.$metadata?.httpStatusCode === 412 ||
          err.name === "PreconditionFailed")
      ) {
        // Another worker created the object first
        return { acquired: false, epoch: nextEpoch };
      }
      throw err;
    }
  }
 
  /** Gracefully releases the lock by marking it as expired. */
  async releaseLock(epoch: number): Promise<void> {
    const lockKey =
      `${this.lockPrefix}${String(epoch).padStart(10, "0")}.json`;
    const payload: LockPayload = {
      leaderId: this.leaderId,
      validityMs: 0,
      expired: true,
    };
 
    await this.s3.send(
      new PutObjectCommand({
        Bucket: this.bucket,
        Key: lockKey,
        Body: JSON.stringify(payload),
        ContentType: "application/json",
      }),
    );
  }
}

Trade-offs: Clocks, Pauses, and Fencing

Relying on time-based lease expirations introduces the classic distributed systems caveats.

Clock Skew

Non-Atomic Execution and Process Pauses

Even with perfectly synchronized clocks, checking lock validity and performing the work is never atomic. A worker can experience an unannounced pause — a GC stop-the-world pause, a page fault, a network delay — after verifying its lock is valid but before writing to storage. By the time it resumes, its lease has expired, another node has claimed leadership, and the stale worker's write causes a lost update.

Fencing Off Zombie Writes

Because time-based locks alone cannot guarantee safety against pauses and drift, downstream storage or APIs should enforce fencing tokens:

Conclusion

A distributed lock used to mean ZooKeeper, DynamoDB, or a queue-service hack. But strong read-after-write consistency (2020) plus conditional writes and CAS primitives (2024) changed what S3 is: the storage service itself can now act as the lock manager. Acquiring leadership is a single conditional PutObject; losing the race is a clean 412; fencing comes free with the epoch number. The same two primitives that enabled databases on S3 make a distributed lock as simple as this.

References
  1. Gunnar Morling — Leader Election with S3 Conditional Writes
  2. Martin Kleppmann — How to Do Distributed Locking