Design a File Storage Service (Dropbox)

We’re designing a cloud file storage service like Dropbox or Google Drive. Users store files in the cloud, and those files sync automatically across all their devices. Edit a file on your laptop, and it appears on your phone seconds later.

The core challenge: how do we sync files across devices efficiently, handle conflicts when two people edit the same file, and avoid storing duplicate data when millions of users upload the same file? Let’s figure it out.

Step 1: Requirements

Functional Requirements

• Upload, download, and delete files
• Automatic sync across all devices linked to an account
• File versioning — ability to view and restore previous versions
• Shared folders (multiple users can access the same files)
• Offline editing — changes sync when the device comes back online
• Notifications when a file is changed by someone else

Non-Functional Requirements

• Reliability — files must never be lost. Zero data loss. Period.
• Sync speed — changes should sync in < 5 seconds on a decent connection
• Bandwidth efficiency — only transfer what changed, not the entire file
• Consistency — all devices should eventually see the same file state
• Scale — 500M users, 10M DAU, average 2 GB stored per user
• Large file support — handle files up to 50 GB

Step 2: Estimation

Assumptions:

• 500M total users, 10M daily active users
• Average storage per user: 2 GB
• Average file size: 500 KB
• Each DAU syncs ~10 file changes per day
• 20% of changes are to large files (> 10 MB)

QPS:

File operations:    10M DAU × 10 changes/day = 100M operations/day
Write QPS:          100M / 86,400 ≈ ~1,200 ops/sec
Peak QPS:           ~3,000 ops/sec

Storage:

Total storage:      500M users × 2 GB = 1 EB (1,000 PB)
Daily new uploads:  100M operations × 500 KB avg = 50 TB/day
With deduplication: ~15 TB/day (many users upload similar files)
Versioning overhead: ~20% extra storage for keeping old versions

Bandwidth:

Upload:   50 TB/day / 86,400 = ~600 MB/sec
Download: 3× upload (people download more than they upload) ≈ ~1.8 GB/sec

The big number here is total storage: 1 exabyte. That’s why deduplication is so critical — it can save us 50-70% of storage costs.

Step 3: High-Level Design

How the pieces fit together:

• Desktop Client — the local agent running on the user’s machine. It watches the sync folder for changes, splits files into chunks, computes hashes, and communicates with the server.
• Sync Service — the brain. It knows the latest version of every file, tracks which chunks make up each file, and coordinates updates.
• Chunking Service — splits files into fixed-size chunks and deduplicates them. More on this in the deep dives.
• Block Storage — where the actual file chunks live. Could be S3 or a custom storage system (Dropbox built their own called Magic Pocket).
• Metadata DB — stores everything about the files (names, paths, sizes, versions, who owns them) but NOT the file data itself.
• Notification Service — tells other devices that something changed so they can sync.

Step 4: API Design

POST /api/v1/files/upload/init
  Body: { "file_name": "report.pdf", "file_size": 52428800,
          "parent_folder_id": "folder_42",
          "chunk_hashes": ["abc123", "def456", "ghi789"] }
  Response: { "file_id": "file_99", "upload_id": "up_555",
              "chunks_needed": ["def456", "ghi789"],   -- abc123 already exists!
              "upload_urls": { "def456": "https://s3.../presigned", ... } }

PUT /api/v1/files/upload/{upload_id}/chunk/{chunk_hash}
  Body: <binary chunk data>
  Response: { "status": "ok" }

POST /api/v1/files/upload/{upload_id}/complete
  Response: { "file_id": "file_99", "version": 3 }

GET /api/v1/files/{file_id}
  → Returns file metadata + download URLs for chunks

GET /api/v1/files/{file_id}/versions
  → Returns list of all versions

GET /api/v1/sync?cursor=<last_sync_token>
  → Returns all changes since the last sync point
  Response: {
    "changes": [
      { "type": "modified", "file_id": "file_99", "version": 3, ... },
      { "type": "deleted", "file_id": "file_50", ... }
    ],
    "cursor": "sync_token_abc",
    "has_more": false
  }

POST /api/v1/shares
  Body: { "file_id": "file_99", "user_email": "bob@example.com",
          "permission": "edit" }

The key insight in the upload API: Notice how the client sends chunk hashes before uploading any data. The server checks which chunks it already has (from other users or previous versions). If a chunk already exists, we skip the upload. This is deduplication in action — the client might only need to upload 1 out of 10 chunks for an edited file.

Step 5: Data Model

-- Users table (PostgreSQL)
CREATE TABLE users (
    user_id         BIGINT PRIMARY KEY,
    email           VARCHAR(255) UNIQUE,
    name            VARCHAR(100),
    storage_used    BIGINT DEFAULT 0,        -- bytes
    storage_limit   BIGINT DEFAULT 2147483648, -- 2 GB free tier
    created_at      TIMESTAMP
);

-- Files table (metadata only, PostgreSQL)
CREATE TABLE files (
    file_id         BIGINT PRIMARY KEY,
    owner_id        BIGINT NOT NULL,
    parent_folder_id BIGINT,                 -- null for root
    file_name       VARCHAR(255),
    is_folder       BOOLEAN DEFAULT FALSE,
    latest_version  INT DEFAULT 1,
    file_size       BIGINT,
    mime_type       VARCHAR(100),
    is_deleted      BOOLEAN DEFAULT FALSE,   -- soft delete
    created_at      TIMESTAMP,
    updated_at      TIMESTAMP,
    INDEX idx_parent (parent_folder_id, file_name),
    INDEX idx_owner (owner_id)
);

-- File versions (PostgreSQL)
CREATE TABLE file_versions (
    file_id         BIGINT,
    version         INT,
    file_size       BIGINT,
    chunk_hashes    TEXT[],                  -- ordered list of chunk hashes
    modified_by     BIGINT,
    created_at      TIMESTAMP,
    PRIMARY KEY (file_id, version)
);

-- Chunks (PostgreSQL — metadata. Actual data in block storage)
CREATE TABLE chunks (
    chunk_hash      VARCHAR(64) PRIMARY KEY, -- SHA-256 hash
    size            INT,
    reference_count INT DEFAULT 1,           -- for garbage collection
    storage_path    TEXT,                    -- path in block storage
    created_at      TIMESTAMP
);

-- Workspaces / shared folders (PostgreSQL)
CREATE TABLE workspace_members (
    workspace_id    BIGINT,
    user_id         BIGINT,
    permission      VARCHAR(10),             -- 'owner', 'edit', 'view'
    created_at      TIMESTAMP,
    PRIMARY KEY (workspace_id, user_id)
);

-- Sync history (for cursor-based sync)
CREATE TABLE sync_log (
    log_id          BIGINT PRIMARY KEY,      -- monotonically increasing
    user_id         BIGINT NOT NULL,
    file_id         BIGINT NOT NULL,
    action          VARCHAR(10),             -- 'create', 'modify', 'delete', 'move'
    version         INT,
    timestamp       TIMESTAMP,
    INDEX idx_user_sync (user_id, log_id)
);

Why track chunks separately? Because the same chunk can appear in millions of files across different users. We store the chunk data once and keep a reference count. When no file references a chunk anymore, we garbage-collect it.

Step 6: Deep Dives

Deep Dive 1: File Chunking and Deduplication

This is the most important optimization in a file storage system. Without it, storage costs would be astronomical.

How chunking works:

Instead of storing a file as one big blob, we split it into fixed-size chunks (typically 4 MB each). Each chunk gets a SHA-256 hash computed from its content.

report.pdf (50 MB)
  → Chunk 1: bytes[0..4MB]     → hash: "abc123"
  → Chunk 2: bytes[4MB..8MB]   → hash: "def456"
  → Chunk 3: bytes[8MB..12MB]  → hash: "ghi789"
  ... (13 chunks total)

The file is stored as: [abc123, def456, ghi789, ...]

Why chunking saves bandwidth:

Imagine we edit a 50 MB file and only change a few paragraphs in the middle. Without chunking, we’d have to re-upload the entire 50 MB. With chunking, only the chunks that changed need to be re-uploaded. If we changed text in chunk 5, we upload one 4 MB chunk instead of 50 MB.

Original file: [abc, def, ghi, jkl, mno, pqr]
Edited file:   [abc, def, ghi, jkl, XYZ, pqr]
                                     ^^^
                              Only this chunk changed!

Upload needed: just 4 MB (the new chunk) instead of 50 MB

How deduplication works:

Here’s where it gets really clever. Since chunks are identified by their content hash (SHA-256), two identical chunks produce the same hash — regardless of who uploaded them.

User A uploads vacation_photos.zip → chunks: [aaa, bbb, ccc]
User B uploads the SAME file       → chunks: [aaa, bbb, ccc]

When User B tries to upload, the server says:
"I already have chunks aaa, bbb, and ccc. Upload nothing."

We just saved 100% of the storage and bandwidth for User B's upload.

This is called content-addressable storage. The address (hash) is derived from the content. Same content = same address = stored only once.

Dedup in practice saves a ton:

• Email attachments that millions of people receive (same PDF, same spreadsheet)
• Common libraries and frameworks (how many copies of React exist in Dropbox?)
• OS files and application binaries

Dropbox reported that deduplication saved them 75% of their storage.

One important consideration: We can also use variable-length chunking (rolling hash / Rabin fingerprint) instead of fixed 4 MB chunks. This handles file insertions better — if we insert text at the beginning of a file, fixed-size chunks would shift everything and all chunks would change. Variable-length chunking is smarter about finding natural boundaries. But fixed-size is simpler and works well enough for most cases.

Deep Dive 2: Sync Conflict Resolution

What happens when two people edit the same file at the same time? Or when we edit a file offline on two different devices?

The problem:

Device A (offline): edits report.pdf → creates version 3
Device B (offline): edits report.pdf → creates version 3

Both come online. Which version 3 wins?

Option A: Last Writer Wins (LWW)

The simplest approach. Whichever device syncs last overwrites the other. The “losing” version gets saved as a previous version so nothing is lost.

Device A syncs at 10:00:01 → version 3 saved
Device B syncs at 10:00:05 → version 4 (overwrites, but version 3 is kept in history)

This is what Dropbox does for regular files. If there’s a conflict, Dropbox saves the conflicting copy as “report (Manish’s conflicted copy).pdf” right next to the original. The user resolves it manually.

Option B: Operational Transform (OT) / CRDTs

For real-time collaborative editing (like Google Docs), we need something smarter. Instead of saving the whole file, we track individual operations (“insert ‘hello’ at position 42”, “delete characters 10-15”).

These operations can be transformed to work together even when they arrive out of order. This is how Google Docs lets 10 people type in the same document at the same time.

But OT is incredibly complex to implement. For a file storage system (not a real-time editor), LWW with conflict copies is the pragmatic choice.

Our approach for the interview:

1. Every file has a version number in the metadata DB
2. When a client uploads changes, it includes the version number it’s based on
3. If the server’s current version matches, the update succeeds (optimistic concurrency)
4. If someone else updated the file in between, we have a conflict
5. We save the conflicting version as a separate file and notify the user
6. The user resolves the conflict manually

Client A: "Update file_99, I'm based on version 2"
Server:   "Current version is 2. Accepted. Now version 3."

Client B: "Update file_99, I'm based on version 2"
Server:   "Current version is 3, not 2. Conflict!"
Server:   Save Client B's version as "report (conflict).pdf"
Server:   Notify Client B about the conflict

Deep Dive 3: Real-Time Sync Notification

When a file changes on one device, how do all other devices find out? We need a notification system that triggers sync.

Long polling approach (what Dropbox originally used):

Each device maintains a long-poll connection to the Notification Service. The connection stays open for up to 60 seconds. If a change happens during that time, the server responds immediately. If nothing changes, the connection times out and the client opens a new one.

Desktop Client → Notification Service: "Any changes since sync_token_42?"
                 (connection stays open)

... 20 seconds later, another device uploads a change ...

Notification Service → Desktop Client: "Yes! Changes detected."
Desktop Client → Sync Service: "GET /sync?cursor=sync_token_42"
                → Gets list of changed files
                → Downloads changed chunks
                → Updates local files

WebSocket approach (more modern):

A persistent WebSocket connection between each client and the Notification Service. When a change happens, the server pushes immediately. Lower latency than long polling.

The full sync flow:

1. User A edits report.pdf on their laptop
2. Desktop client detects the change (filesystem watcher)
3. Client chunks the file, computes hashes, uploads new chunks
4. Client calls /upload/complete → Sync Service updates metadata
5. Sync Service writes to the sync log and publishes an event to the message queue
6. Notification Service reads the event
7. Notification Service pushes to User A’s other devices + any shared workspace members
8. Each notified device calls /sync?cursor=... to get the change details
9. Each device downloads the new chunks and reassembles the file locally

Scaling the notification service:

• Each active device maintains one connection. 10M DAU with 2.5 devices average = 25M connections.
• Shard by user_id — each notification server handles a subset of users
• Use a connection registry (Redis) to map user_id to notification server
• The message queue (Kafka) decouples the sync service from notifications — the sync service publishes the event, and however many notification servers consume it

Step 7: Scaling

Block storage:

• Use S3 or a custom object store (Dropbox built Magic Pocket to move off S3 and save costs)
• Replicate across 3+ data centers for durability
• Use storage tiers: hot (frequently accessed), warm (less frequent), cold (old versions)
• Content-addressable storage means dedup is automatic — same hash, same object

Metadata DB:

• Shard by user_id — all of a user’s files live on the same shard
• Read replicas for the sync endpoint (reads vastly outnumber writes)
• The files table is the most critical table — it must be consistent and durable
• Index on (parent_folder_id, file_name) for directory listings

Sync performance:

• The sync endpoint uses cursor-based pagination on the sync_log table
• Clients only request changes since their last sync point — not the full file tree
• For the initial sync (new device), send the full file tree in batches

Chunking optimization:

• 4 MB chunk size is a good balance: small enough to limit re-upload on edits, large enough to avoid too many chunks per file
• Compute chunk hashes on the client side — this avoids uploading data the server already has
• Use SHA-256 for chunk hashing — collision probability is effectively zero

CDN for downloads:

• Popular shared files (company-wide documents) can be served from CDN
• Most file access is personal (user’s own files), so CDN helps less than in video streaming
• CDN is more useful for the web client’s static assets

Rate limiting and quotas:

• Storage quotas per user (2 GB free, 2 TB paid)
• Rate limit file operations per user (prevent abuse)
• Throttle sync for clients that are doing massive initial uploads

Multi-region:

• Store files in the region closest to the user
• For shared workspaces with users in different regions, replicate data to both regions
• Metadata must be globally consistent — use a global metadata store with cross-region replication

In simple language, a file storage service is built on three key ideas. First, chunking — split files into 4 MB pieces so we only transfer what changed. Second, deduplication — store each unique chunk exactly once by using content hashes as identifiers. Third, sync notifications — use WebSocket or long polling to tell devices when something changed, then let the device fetch the changes. The metadata DB (which chunks make up which files) is the source of truth. The block storage (the actual bytes) is the heavy storage layer. And the notification service is the glue that keeps everything in sync across devices.