File Storage Architecture¶

This document describes the file storage system architecture for the OpenAI Files API compatibility layer, including persistent metadata storage.

Table of Contents¶

• Overview
• Storage Modes
• Architecture
• Storage Structure
• Metadata Schema
• Storage Backends
• Content Streaming and the Memory Model
• Authentication and Authorization
• Startup Recovery
• Orphan Detection and Cleanup
• Configuration
• API Endpoints
• Design Decisions

Overview¶

The File Storage system provides OpenAI Files API compatible file management with persistent metadata storage. It allows users to upload files for fine-tuning, batch processing, and other purposes while ensuring data durability across server restarts.

Key Features¶

• OpenAI Files API Compatibility: Full support for /v1/files endpoints
• Persistent Metadata: File metadata survives server restarts
• Automatic Recovery: Rebuilds metadata index from sidecar files on startup
• Orphan Management: Detects and cleans up inconsistent file states
• Pluggable Backends: Support for memory and persistent storage backends
• Streamed Content: Uploads and downloads move chunk by chunk, so peak memory per transfer is a buffer rather than a file

Storage Modes¶

Two metadata storage backends are available via files.metadata_storage: memory and persistent (the default). The diagrams below show why persistent is the default.

In-Memory Metadata (metadata_storage: memory)¶

With the memory backend, file metadata lives only in an in-memory DashMap:

Constraints of the memory backend: - Complete metadata loss on server restart - Orphaned files on disk with no API access - Inconsistent state between files and metadata - No way to recover uploaded files

Persistent Metadata (metadata_storage: persistent, default)¶

With persistent metadata storage:

Architecture¶

Component Diagram¶

Layer Responsibilities¶

Storage Structure¶

Directory Layout¶

storage/
├── a1b2c/                          # Subdirectory (first 5 chars of file ID)
│   ├── file-a1b2c3d4e5f6.bin       # Binary data file
│   └── file-a1b2c3d4e5f6.meta.json # Metadata sidecar file
├── x9y8z/
│   ├── file-x9y8z7w6v5u4.bin
│   └── file-x9y8z7w6v5u4.meta.json
└── ...

File Naming Convention¶

Uniqueness of the .tmp path comes from {id}, which is file- plus a v4 UUID, the same thing that makes the destination .bin path unique. The pid and the per-process sequence number are diagnostic: they let a leftover temporary be traced back to the process and the upload that abandoned it. They are not a uniqueness mechanism, and must not be treated as one, because two containers each see pid 1 with the sequence starting at 0.

The .tmp suffix is deliberately outside the .bin / .meta.json naming the orphan scanner matches, so a live upload is never mistaken for stored content.

Sidecar Pattern Benefits¶

1. Co-location: Data and metadata are stored together
2. Atomic Operations: Metadata writes use atomic rename pattern
3. Easy Backup: Simple directory copy preserves everything
4. Debug Friendly: Human-readable JSON metadata
5. No External Dependencies: No database required

Metadata Schema¶

FileMetadata Structure¶

{
  "id": "file-abc123def456",
  "object": "file",
  "filename": "training_data.jsonl",
  "bytes": 1048576,
  "purpose": "fine-tune",
  "created_at": 1699574400,
  "content_type": "application/jsonl",
  "storage_path": "a1b2c/file-abc123def456.bin"
}

Field Descriptions¶

Supported Purposes¶

Storage Backends¶

MetadataBackend Trait¶

#[async_trait]
pub trait MetadataBackend: Send + Sync {
    async fn insert(&self, metadata: FileMetadata) -> Result<(), FileError>;
    async fn get(&self, id: &str) -> Option<FileMetadata>;
    async fn remove(&self, id: &str) -> Option<FileMetadata>;
    async fn list(&self, query: &FileListQuery) -> Vec<FileMetadata>;
    async fn len(&self) -> usize;
    async fn is_empty(&self) -> bool;
}

Backend Comparison¶

Write Path (Persistent)¶

1. Generate file ID
2. Stream content into storage/{subdir}/.file-{id}.{pid}.{seq}.tmp
3. Validate (size, type, content) once the purpose is known
4. fsync + atomic rename: .file-{id}.{pid}.{seq}.tmp → file-{id}.bin
5. Create metadata JSON
6. Write to temp file: file-{id}.meta.json.tmp
7. Atomic rename: file-{id}.meta.json.tmp → file-{id}.meta.json
8. Update in-memory cache

Steps 2 to 4 are what make the data file atomic: the total size is not known when the first byte arrives, so the content cannot be written straight to its final path. A reader therefore never observes a half-written .bin, and an upload that fails validation, errors, or is cancelled mid-flight leaves nothing behind. See Content Streaming and the Memory Model.

Read Path (Persistent)¶

1. Check in-memory cache (DashMap)
2. If hit → return cached metadata
3. If miss → (only on startup recovery)
   a. Scan directory for .meta.json files
   b. Parse and validate each file
   c. Populate cache

Content Streaming and the Memory Model¶

File content is streamed end to end. An upload is written to disk chunk by chunk as it arrives off the wire, and a download is read back chunk by chunk as it is written to the client. Peak resident memory for a transfer is therefore a 64KB buffer plus whatever the HTTP layer holds, not the size of the file.

This matters for capacity planning: files.max_file_size is a policy and disk decision, not a memory decision. A router with a 512Mi memory limit can serve a 512MB upload, and concurrent transfers add a chunk buffer each rather than a file each.

Upload Path¶

multipart chunk --> size check --> incremental validation --> temp file
                                                                  |
                       (all checks passed) --- fsync + rename ----+--> file-{id}.bin

1. Size check. Each chunk is counted before it is accepted. Crossing files.max_file_size returns 413 immediately, without reading the rest of the body.
2. Incremental validation. The first 13 bytes are retained for the magic-byte checks (image signatures for vision, executable signatures for assistants / user_data), and a non-allocating UTF-8 validator tracks the whole body for the JSONL purposes. The validator carries a partial multi-byte sequence across a chunk boundary, so a character split between two network chunks is judged exactly as it would be in a single buffer, and a body that ends mid-sequence is rejected.
3. Verdict. A multipart client may send the file part before the purpose part, and the purpose selects which check applies, so the verdict is taken after the body is complete. Nothing is published before it passes.
4. Publish. The temporary file is fsynced and renamed into place. Only then is the metadata sidecar written.

Download Path¶

The authorization decision is made against metadata alone, before any content is read. A caller who knows a file id but does not own it gets 403 without the router ever opening the file. The body is then a stream over the open file, so the response starts before the file has been fully read and never exists in memory as a whole.

What Still Buffers¶

Two consumers hold a whole file by nature, and both are bounded independently of files.max_file_size:

Because of these, the router still emits a startup warning when files.max_file_size exceeds a quarter of the container memory limit reported by cgroup v2 (/sys/fs/cgroup/memory.max). The warning is skipped when there is no cgroup v2 limit, when the limit reads max, and on platforms without cgroups (macOS, bare metal), so it never fires spuriously.

Transient Disk Use¶

The verdict is taken after the whole body is on disk, because the purpose arrives in a later multipart part and the purpose selects the check. A rejected upload therefore occupies its full size on disk for the duration of the transfer before it is discarded. With N concurrent uploads the transient peak is N times max_file_size, and the Files group carries no rate-limit mount of its own.

This is a deliberate trade: disk pressure degrades a request, where the buffered design it replaced could take the whole process down with an OOM kill. Operators who want the peak bounded have three levers:

• server.max_concurrent_requests caps in-flight requests process-wide. It is unset by default.
• A smaller files.max_file_size lowers the per-upload ceiling directly.
• A dedicated volume for files.storage_path keeps a full upload directory from affecting anything else on the node.

Failure Modes¶

The 24 hour threshold exists so the sweep cannot delete a temporary belonging to an upload that is still streaming, including one in another router process sharing the storage directory. A temporary's mtime advances with every write, so an in-progress transfer keeps refreshing itself out of range no matter how long it runs. The reclamation is deliberately not gated on cleanup_orphans_on_startup: that flag guards deleting .bin files that may still be recoverable data, whereas an aged temporary is unambiguously garbage that no API can reach.

Authentication and Authorization¶

The Files API includes authentication and authorization to secure file operations.

Authentication Methods¶

Authorization Model¶

┌─────────────────────────────────────────────────────────────────┐
│                    Files API Request                             │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Authentication Layer                           │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Extract Bearer Token → Validate API Key → Check Scope      ││
│  └─────────────────────────────────────────────────────────────┘│
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Authorization Layer                            │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Check File Ownership → Admin Override → Allow/Deny         ││
│  └─────────────────────────────────────────────────────────────┘│
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                      File Operation                              │
└─────────────────────────────────────────────────────────────────┘

File Ownership¶

When enforce_ownership is enabled (default):

Metadata Fields for Authorization¶

The FileMetadata structure includes ownership fields:

{
  "id": "file-abc123def456",
  "owner_id": "user-xyz789",
  "organization_id": "org-abc123",
  "source_ip": "192.168.1.100",
  "created_at": 1699574400
}

Audit Logging¶

All file operations are logged with authentication context:

INFO file_uploaded file_id="file-abc123" user_id="user-xyz" org_id="org-abc" client_ip="192.168.1.1"
INFO file_downloaded file_id="file-abc123" user_id="user-xyz"
INFO file_deleted file_id="file-abc123" user_id="user-xyz" client_ip="192.168.1.1"
WARN file_access_denied file_id="file-abc123" user_id="user-xyz" file_owner="user-other"

Security Considerations¶

1. Development Keys: Only available when CONTINUUM_DEV_MODE is set or in debug builds
2. Scope Requirements: API keys must have the configured scope (default: "files")
3. Legacy Files: Files without owner_id are accessible by all authenticated users
4. Admin Override: Users with "admin" scope bypass ownership checks if configured

Startup Recovery¶

Recovery Process¶

┌─────────────────────────────────────────────────────┐
│                 Server Startup                       │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│         Scan storage directory recursively          │
│         Find all *.meta.json files                  │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│         For each .meta.json file:                   │
│         1. Parse JSON content                       │
│         2. Validate schema                          │
│         3. Check corresponding .bin exists          │
│         4. Add to in-memory cache                   │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│         Log recovery statistics:                    │
│         - Files recovered: N                        │
│         - Orphans detected: M                       │
└─────────────────────────────────────────────────────┘

Recovery Guarantees¶

1. Idempotent: Safe to run multiple times
2. Non-destructive: Never deletes files during recovery
3. Partial Success: Continues even if some files are corrupted
4. Logged: All recovery actions are logged for debugging

Orphan Detection and Cleanup¶

Orphan Types¶

Detection Algorithm¶

pub async fn detect_orphans(&self) -> Result<(Vec<PathBuf>, Vec<PathBuf>), FileError> {
    // Scan all files in storage directory
    for file in storage_directory {
        if file.ends_with(".bin") {
            // Check if corresponding .meta.json exists
            let meta_path = file.replace(".bin", ".meta.json");
            if !meta_path.exists() {
                orphaned_data.push(file);
            }
        } else if file.ends_with(".meta.json") {
            // Check if corresponding .bin exists
            let data_path = file.replace(".meta.json", ".bin");
            if !data_path.exists() {
                orphaned_metadata.push(file);
            }
        }
    }
    Ok((orphaned_data, orphaned_metadata))
}

Cleanup Options¶

Warning: Data file cleanup requires manual intervention to prevent accidental data loss.

TOCTOU Safety Note¶

Orphan cleanup is NOT safe during active file operations due to Time-of-Check-Time-of-Use race conditions:

Thread A: detect_orphans() → finds file-X as orphan
Thread B: upload() → creates metadata for file-X
Thread A: cleanup() → deletes "orphan" (now valid!)

Recommendation: Only run cleanup during server startup or maintenance windows.

Configuration¶

YAML Configuration¶

files:
  enabled: true
  max_file_size: 536870912        # 512MB
  storage_path: "./data/files"    # Supports ~ expansion
  retention_days: 0               # startup sweep in days; 0 = keep forever
  metadata_storage: persistent    # "memory" or "persistent"
  cleanup_orphans_on_startup: false

max_file_size supports 1KB to 5GB, and the whole range takes effect: the outer HTTP body cap on the upload routes is derived from this value (plus a fixed 1MB allowance for the multipart envelope), so raising it above 512MB genuinely raises the limit.

When retention_days is nonzero, startup runs a retention sweep over stored metadata records. The age basis is the metadata sidecar created_at timestamp, not filesystem mtime; files whose age is at least the configured number of days are removed with their .meta.json sidecar. retention_days: 0 keeps files forever. The sweep removes metadata before content, so new lookups fail closed once a file is selected; a download that already opened the content handle keeps streaming on POSIX, while a caller that resolved metadata but has not opened content yet may see a 404 if the unlink wins that race. Each nonzero sweep logs the deleted and failed counts and increments file_retention_deletes_total{status="success"} for reclaimed files.

A value outside that range is reported as a startup warning, not a startup failure, and the derived body cap is clamped into the range. The warning rather than a hard error is deliberate: max_file_size: 0 is accepted today, small values are plausible, and a fatal check would turn a working router into one that refuses to start. The clamp is what actually protects the routes, so an out-of-range value can neither remove the outer cap nor push it below the multipart envelope.

There are two places an oversized upload can be stopped, and both answer the same way:

The second case used to answer 400 invalid_request, because the handlers classified every multipart failure as malformed input. A body the transport truncated is a size failure, so the status now follows the cause rather than how the client happened to frame the request.

Environment overrides¶

The four files.auth fields have dedicated direct CONTINUUM_FILES_* overrides; no other Files API field does, and none of them expand ${VAR} references, so render deployment-specific values for the rest of files: into YAML/TOML before validation, or maintain separate validated deployment files.

export CONTINUUM_FILES_AUTH_METHOD=none                  # or api_key
export CONTINUUM_FILES_AUTH_SCOPE=files
export CONTINUUM_FILES_ENFORCE_OWNERSHIP=true             # strict "true"/"false" only
export CONTINUUM_FILES_ADMIN_ACCESS_ALL=true               # strict "true"/"false" only

Each variable applies even when the configuration has no files: section at all; the override is layered onto FilesConfig::default() in that case, matching how the rest of the router treats an absent files: section. These variables have no CLI-flag counterpart, so precedence is simply the environment variable (when set) over the file's files.auth value over the default. An unrecognized CONTINUUM_FILES_AUTH_METHOD value, or a boolean variable set to anything other than exactly true or false, fails startup with a clear error instead of silently falling back.

The config-file watcher re-reads the process environment on every reload, so a CONTINUUM_FILES_* value is never silently replaced by file contents when the file changes; it is re-applied on top of the freshly parsed file every time.

Storage Backend Selection¶

API Endpoints¶

POST /v1/files¶

Upload a new file. The body is streamed to storage as it arrives; see Content Streaming and the Memory Model.

curl -X POST http://localhost:8080/v1/files \
  -H "Content-Type: multipart/form-data" \
  -F "file=@training.jsonl" \
  -F "purpose=fine-tune"

GET /v1/files¶

List all files.

curl http://localhost:8080/v1/files?purpose=fine-tune&limit=10

GET /v1/files/:id¶

Get file metadata.

curl http://localhost:8080/v1/files/file-abc123

GET /v1/files/:id/content¶

Download file content. Authorization is decided before the file is opened, and the response body is streamed from disk.

curl http://localhost:8080/v1/files/file-abc123/content -o downloaded.jsonl

DELETE /v1/files/:id¶

Delete a file.

curl -X DELETE http://localhost:8080/v1/files/file-abc123

Design Decisions¶

Why Sidecar JSON Files?¶

Alternatives Considered:

Decision: Sidecar JSON files were chosen because: 1. Fits existing file-based architecture 2. No additional dependencies (uses existing serde_json) 3. Files and metadata are co-located for easy backup/restore 4. Atomic writes possible with rename pattern 5. Human-readable for debugging

Why In-Memory Cache + Disk?¶

Pattern: Write-through cache with disk persistence

Write: Cache → Disk (synchronous)
Read:  Cache (fast path)

Benefits: - Sub-millisecond read latency - Durable writes - Automatic recovery on restart

Why Not Database?¶

For the Files API use case: - Typically hundreds to thousands of files, not millions - Simple key-value access pattern - No complex queries needed - Filesystem already provides atomicity guarantees

A database would add: - Operational complexity - Additional dependency - Potential single point of failure

Related Documentation¶

• Configuration Guide - Full configuration reference
• API Documentation - Complete API reference
• Architecture Guide - Overall system architecture