File Storage Architecture¶

This document describes the file storage system architecture for the OpenAI Files API compatibility layer, including persistent metadata storage.

Table of Contents¶

• Overview
• Storage Modes
• Architecture
• Storage Structure
• Metadata Schema
• Storage Backends
• Content Streaming and the Memory Model
• Authentication and Authorization
• Startup Recovery
• Orphan Detection and Cleanup
• Configuration
• API Endpoints
• Design Decisions

Overview¶

The File Storage system provides OpenAI Files API compatible file management with persistent metadata storage. It allows users to upload files for fine-tuning, batch processing, and other purposes while ensuring data durability across server restarts.

Key Features¶

• OpenAI Files API Compatibility: Full support for /v1/files endpoints
• Persistent Metadata: File metadata survives server restarts
• Automatic Recovery: Rebuilds metadata index from sidecar files on startup
• Orphan Management: Detects and cleans up inconsistent file states
• Pluggable Backends: Support for memory and persistent storage backends
• Streamed Content: Uploads and downloads move chunk by chunk, so peak memory per transfer is a buffer rather than a file

Storage Modes¶

Two metadata storage backends are available via files.metadata_storage: memory and persistent (the default). The diagrams below show why persistent is the default.

In-Memory Metadata (metadata_storage: memory)¶

With the memory backend, file metadata lives only in an in-memory DashMap:

Constraints of the memory backend:

• Complete metadata loss on server restart
• Orphaned files on disk with no API access
• Inconsistent state between files and metadata
• No way to recover uploaded files

Persistent Metadata (metadata_storage: persistent, default)¶

With persistent metadata storage:

Architecture¶

Component Diagram¶

Layer Responsibilities¶

Storage Structure¶

Directory Layout¶

storage/
├── a1b2c/                          # Subdirectory (first 5 chars of file ID)
│   ├── file-a1b2c3d4e5f6.bin       # Binary data file
│   └── file-a1b2c3d4e5f6.meta.json # Metadata sidecar file
├── x9y8z/
│   ├── file-x9y8z7w6v5u4.bin
│   └── file-x9y8z7w6v5u4.meta.json
└── ...

File Naming Convention¶

Uniqueness of the .tmp path comes from {id}, which is file- plus a v4 UUID, the same identifier that makes the destination .bin path unique. The process ID and per-process sequence number are diagnostic: they trace a leftover temporary back to the process and upload that abandoned it. They are not a uniqueness mechanism because two containers each see PID 1 with the sequence starting at 0.

The .tmp suffix is deliberately outside the .bin / .meta.json naming that the orphan scanner matches, so a live upload is never mistaken for stored content.

Sidecar Pattern Benefits¶

1. Co-location: Data and metadata are stored together
2. Atomic Operations: Metadata writes use atomic rename pattern
3. Easy Backup: Simple directory copy preserves everything
4. Debug Friendly: Human-readable JSON metadata
5. No External Dependencies: No database required

Metadata Schema¶

FileMetadata Structure¶

{
  "id": "file-abc123def456",
  "object": "file",
  "filename": "training_data.jsonl",
  "bytes": 1048576,
  "purpose": "fine-tune",
  "created_at": 1699574400,
  "content_type": "application/jsonl",
  "storage_path": "a1b2c/file-abc123def456.bin"
}

Field Descriptions¶

Supported Purposes¶

Storage Backends¶

MetadataBackend Trait¶

#[async_trait]
pub trait MetadataBackend: Send + Sync {
    async fn insert(&self, metadata: FileMetadata) -> Result<(), FileError>;
    async fn get(&self, id: &str) -> Option<FileMetadata>;
    async fn remove(&self, id: &str) -> Option<FileMetadata>;
    async fn list(&self, query: &FileListQuery) -> Vec<FileMetadata>;
    async fn list_page(&self, query: &FileListQuery) -> FileListPage;
    async fn list_by_owner(&self, query: &FileListQuery, owner_id: &str) -> FileListPage;
    async fn len(&self) -> usize;
    async fn is_empty(&self) -> bool;
}

Listing goes through list_page (all owners) and list_by_owner (a non-admin caller under files.auth.enforce_ownership). Both stores filter by purpose and owner while iterating their in-memory index, then one shared function (src/services/files/listing.rs) sorts by created_at with the file ID as a tie-break, applies the after cursor, cuts to limit, and sets has_more.

Filtering before the page cut keeps a caller's page from being filled by other owners' newer files. An after ID outside the filtered listing returns an empty page with has_more: false. Both methods have default bodies built on list, so an implementor that provides only list still pages correctly.

Backend Comparison¶

Write Path (Persistent)¶

1. Generate file ID
2. Stream content into storage/{subdir}/.file-{id}.{pid}.{seq}.tmp
3. Validate (size, type, content) once the purpose is known
4. fsync + atomic rename: .file-{id}.{pid}.{seq}.tmp → file-{id}.bin
5. Create metadata JSON
6. Write to temp file: file-{id}.meta.json.tmp
7. Atomic rename: file-{id}.meta.json.tmp → file-{id}.meta.json
8. Update in-memory cache

Steps 2 to 4 make the data file atomic: the total size is not known when the first byte arrives, so the content cannot be written straight to its final path. A reader therefore never observes a half-written .bin, and an upload that fails validation, errors, or is cancelled mid-flight leaves nothing behind. See Content Streaming and the Memory Model.

Read Path (Persistent)¶

1. Check in-memory cache (DashMap)
2. If hit → return cached metadata
3. If miss → (only on startup recovery)
   a. Scan directory for .meta.json files
   b. Parse and validate each file
   c. Populate cache

Content Streaming and the Memory Model¶

File content is streamed end to end. An upload is written to disk chunk by chunk as it arrives off the wire, and a download is read back chunk by chunk as it is written to the client. Peak resident memory for a transfer is therefore a 64KB buffer plus whatever the HTTP layer holds, not the size of the file.

This matters for capacity planning: files.max_file_size is a policy and disk decision, not a memory decision. A router with a 512Mi memory limit can serve a 512MB upload, and concurrent transfers add a chunk buffer each rather than a file each.

Upload Path¶

multipart chunk --> size check --> incremental validation --> temp file
                                                                  |
                       (all checks passed) --- fsync + rename ----+--> file-{id}.bin

1. Size check. Each chunk is counted before it is accepted. Crossing files.max_file_size returns 413 immediately, without reading the rest of the body.
2. Incremental validation. The first 13 bytes are retained for magic-byte checks (image signatures for vision, executable signatures for assistants / user_data), and a non-allocating UTF-8 validator tracks the whole body for JSONL purposes. The validator carries a partial multi-byte sequence across a chunk boundary so a character split between two network chunks is judged identically to a single buffer, and a body ending mid-sequence is rejected.
3. Verdict. A multipart client may send the file part before the purpose part, and the purpose selects which check applies, so the verdict is taken after the body is complete. Nothing is published before it passes.
4. Publish. The temporary file is fsynced and renamed into place. Only then is the metadata sidecar written.

Download Path¶

The authorization decision is made against metadata alone, before any content is read. A caller who knows a file ID but does not own it receives 403 without the router opening the file. The body is then streamed over the open file, so the response starts before the file has been fully read and never exists in memory as a whole.

What Still Buffers¶

Two consumers hold a whole file by nature, and both are bounded independently of files.max_file_size:

Because of these paths, the router emits a startup warning when files.max_file_size exceeds a quarter of the container memory limit reported by cgroup v2 (/sys/fs/cgroup/memory.max). The warning is skipped when there is no cgroup v2 limit, when the limit reads max, and on platforms without cgroups (macOS, bare metal).

Transient Disk Use¶

The verdict is taken after the whole body is on disk because the purpose can arrive in a later multipart part and selects the validation check. A rejected upload therefore occupies its full size on disk for the duration of the transfer before it is discarded. With N concurrent uploads the transient peak is N times max_file_size, and the Files group carries no rate-limit mount of its own.

This is a deliberate trade: disk pressure degrades a request, whereas the buffered design it replaced could terminate the process with an OOM kill. Operators who want the peak bounded have three levers:

• server.max_concurrent_requests caps in-flight requests process-wide. It is unset by default.
• A smaller files.max_file_size lowers the per-upload ceiling directly.
• A dedicated volume for files.storage_path keeps a full upload directory from affecting anything else on the node.

Failure Modes¶

When a process is killed mid-upload (SIGKILL, OOM kill, node eviction, or power loss), no in-process cleanup can run, so the .tmp file remains on disk without becoming visible through the API. On startup, the router reclaims its own temporaries (.file-{id}...tmp and file-{id}.meta.json.tmp) older than 24 hours, and FileService::detect_orphans reports the count so leaks remain observable. The pattern match stays narrow because files.storage_path may not be owned exclusively by the router.

The 24-hour threshold prevents the sweep from deleting a temporary belonging to an upload that is still streaming, including one in another router process sharing the storage directory. Because a temporary's mtime advances with every write, an active transfer keeps refreshing itself out of range. This reclamation is not gated on cleanup_orphans_on_startup: that flag guards .bin files that may still be recoverable data, whereas an aged temporary is unreachable by any API.

Authentication and Authorization¶

The Files API includes authentication and authorization to secure file operations.

Authentication Methods¶

Authorization Model¶

┌─────────────────────────────────────────────────────────────────┐
│                    Files API Request                             │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Authentication Layer                           │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Extract Bearer Token → Validate API Key → Check Scope      ││
│  └─────────────────────────────────────────────────────────────┘│
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Authorization Layer                            │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Check File Ownership → Admin Override → Allow/Deny         ││
│  └─────────────────────────────────────────────────────────────┘│
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                      File Operation                              │
└─────────────────────────────────────────────────────────────────┘

File Ownership¶

When enforce_ownership is enabled (default):

A file with no owner was stored before ownership tracking existed and stays readable by every authenticated caller. POST /admin/files does not create such files: it records the synthetic owner __admin__, which no caller identity matches, even a key whose user_id is __admin__. On the public Files API only an admin-scope key with admin_can_access_all reaches an admin upload, and the Responses and Anthropic file resolvers, which have no admin policy, refuse it.

Metadata Fields for Authorization¶

The FileMetadata structure includes ownership fields:

{
  "id": "file-abc123def456",
  "owner_id": "user-xyz789",
  "organization_id": "org-abc123",
  "source_ip": "192.168.1.100",
  "created_at": 1699574400
}

Audit Logging¶

All file operations are logged with authentication context:

INFO file_uploaded file_id="file-abc123" user_id="user-xyz" org_id="org-abc" client_ip="192.168.1.1"
INFO file_downloaded file_id="file-abc123" user_id="user-xyz"
INFO file_deleted file_id="file-abc123" user_id="user-xyz" client_ip="192.168.1.1"
WARN file_access_denied file_id="file-abc123" user_id="user-xyz" file_owner="user-other"

Security Considerations¶

1. Development Keys: Only available when CONTINUUM_DEV_MODE is true or 1, or in debug builds where the variable is unset
2. Scope Requirements: API keys must have the configured scope (default: "files")
3. Legacy Files: Files without owner_id are accessible by all authenticated users
4. Admin Override: Users with "admin" scope bypass ownership checks if configured

Startup Recovery¶

Recovery Process¶

┌─────────────────────────────────────────────────────┐
│                 Server Startup                       │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│         Scan storage directory recursively          │
│         Find all *.meta.json files                  │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│         For each .meta.json file:                   │
│         1. Parse JSON content                       │
│         2. Validate schema                          │
│         3. Check corresponding .bin exists          │
│         4. Add to in-memory cache                   │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│         Log recovery statistics:                    │
│         - Files recovered: N                        │
│         - Orphans detected: M                       │
└─────────────────────────────────────────────────────┘

Recovery Guarantees¶

1. Idempotent: Safe to run multiple times
2. Non-destructive: Never deletes files during recovery
3. Partial Success: Continues even if some files are corrupted
4. Logged: All recovery actions are logged for debugging

Orphan Detection and Cleanup¶

Orphan Types¶

Detection Algorithm¶

pub async fn detect_orphans(&self) -> Result<(Vec<PathBuf>, Vec<PathBuf>), FileError> {
    // Scan all files in storage directory
    for file in storage_directory {
        if file.ends_with(".bin") {
            // Check if corresponding .meta.json exists
            let meta_path = file.replace(".bin", ".meta.json");
            if !meta_path.exists() {
                orphaned_data.push(file);
            }
        } else if file.ends_with(".meta.json") {
            // Check if corresponding .bin exists
            let data_path = file.replace(".meta.json", ".bin");
            if !data_path.exists() {
                orphaned_metadata.push(file);
            }
        }
    }
    Ok((orphaned_data, orphaned_metadata))
}

Cleanup Options¶

Warning: Data file cleanup requires manual intervention to prevent accidental data loss.

TOCTOU Safety Note¶

Orphan cleanup is NOT safe during active file operations due to Time-of-Check-Time-of-Use race conditions:

Thread A: detect_orphans() → finds file-X as orphan
Thread B: upload() → creates metadata for file-X
Thread A: cleanup() → deletes "orphan" (now valid!)

Recommendation: Only run cleanup during server startup or maintenance windows.

Configuration¶

YAML Configuration¶

files:
  enabled: true
  max_file_size: 536870912        # 512MB
  storage_path: "./data/files"    # Supports ~ expansion
  retention_days: 0               # startup sweep in days; 0 = keep forever
  metadata_storage: persistent    # "memory" or "persistent"
  cleanup_orphans_on_startup: false
  auth:
    method: api_key               # "none" or "api_key"
    required_scope: files
    enforce_ownership: true
    admin_can_access_all: true
  openai:
    strategy: base64              # "base64" or "upload"
    fallback_to_base64: true
    cache_capacity: 1000
    cache_ttl_secs: 86400
  rate_limiting:
    enabled: false
    uploads_per_minute: 10        # POST /v1/files
    downloads_per_minute: 100     # GET /v1/files/{id}/content
    operations_per_minute: 1000   # GET /v1/files, GET/HEAD/DELETE /v1/files/{id}

When files.rate_limiting.enabled is true, the router enforces separate per-caller token-bucket limits (in requests per minute) across three operation categories: files_upload (uploads_per_minute), files_download (downloads_per_minute), and files_operations (operations_per_minute).

Each bucket is keyed per caller by the SHA-256 hash of the presented API key (Authorization: Bearer or x-api-key), the authenticated user ID, or the client IP, and returns x-ratelimit-* headers on every Files API response (429 Too Many Requests with Retry-After and details.limit_type when exceeded).

max_file_size supports 1KB to 5GB, and the whole range takes effect: the outer HTTP body cap on the upload routes is derived from this value (plus a fixed 1MB allowance for the multipart envelope), so raising it above 512MB raises the effective limit.

When retention_days is nonzero, startup runs a retention sweep over stored metadata records using the metadata sidecar created_at timestamp rather than filesystem mtime. Files whose age is at least the configured number of days are removed along with their .meta.json sidecar, while retention_days: 0 keeps files forever.

The sweep removes metadata before content so new lookups fail closed once a file is selected. On POSIX, a download that already opened the content handle continues streaming, while a caller that resolved metadata before opening content may receive a 404 if the unlink completes first. Each nonzero sweep logs the deleted and failed counts and increments file_retention_deletes_total{status="success"} for reclaimed files.

A max_file_size value outside the 1KB to 5GB range is reported as a startup warning rather than a startup failure, and the derived body cap is clamped into the range so an out-of-range value can neither remove the outer cap nor push it below the multipart envelope.

There are two places an oversized upload can be stopped, and both answer the same way:

Environment overrides¶

The four files.auth fields have dedicated direct CONTINUUM_FILES_* overrides; no other Files API field does, and none of them expand ${VAR} references, so render deployment-specific values for the rest of files: into YAML/TOML before validation, or maintain separate validated deployment files.

export CONTINUUM_FILES_AUTH_METHOD=none                  # or api_key
export CONTINUUM_FILES_AUTH_SCOPE=files
export CONTINUUM_FILES_ENFORCE_OWNERSHIP=true             # strict "true"/"false" only
export CONTINUUM_FILES_ADMIN_ACCESS_ALL=true               # strict "true"/"false" only

Each variable applies even when the configuration has no files: section, layering onto FilesConfig::default() just like the rest of the router. Because these variables have no CLI-flag counterpart, precedence is environment variable > files.auth file value > default. An unrecognized CONTINUUM_FILES_AUTH_METHOD value, or a boolean variable set to anything other than true or false, fails startup with an explicit error instead of falling back silently.

The config-file watcher re-reads the process environment on every reload, so a CONTINUUM_FILES_* value is never silently replaced by file contents when the file changes; it is re-applied on top of the freshly parsed file every time.

Storage Backend Selection¶

API Endpoints¶

POST /v1/files¶

Upload a new file. The body is streamed to storage as it arrives; see Content Streaming and the Memory Model.

curl -X POST http://localhost:8080/v1/files \
  -H "Content-Type: multipart/form-data" \
  -F "file=@training.jsonl" \
  -F "purpose=fine-tune"

GET /v1/files¶

List all files.

curl http://localhost:8080/v1/files?purpose=fine-tune&limit=10

GET /v1/files/:id¶

Get file metadata.

curl http://localhost:8080/v1/files/file-abc123

GET /v1/files/:id/content¶

Download file content. Authorization is decided before the file is opened, and the response body is streamed from disk.

curl http://localhost:8080/v1/files/file-abc123/content -o downloaded.jsonl

DELETE /v1/files/:id¶

Delete a file.

curl -X DELETE http://localhost:8080/v1/files/file-abc123

Design Decisions¶

Why Sidecar JSON Files?¶

Alternatives Considered:

Decision: Sidecar JSON files were chosen because:

1. Fits existing file-based architecture
2. No additional dependencies (uses existing serde_json)
3. Files and metadata are co-located for easy backup/restore
4. Atomic writes possible with rename pattern
5. Human-readable for debugging

Why In-Memory Cache + Disk?¶

Pattern: Write-through cache with disk persistence

Write: Cache → Disk (synchronous)
Read:  Cache (fast path)

Benefits:

• Sub-millisecond read latency
• Durable writes
• Automatic recovery on restart

Why Not Database?¶

For the Files API use case:

• Typically hundreds to thousands of files, not millions
• Simple key-value access pattern
• No complex queries needed
• Filesystem already provides atomicity guarantees

A database would add:

• Operational complexity
• Additional dependency
• Potential single point of failure

Related Documentation¶

• Configuration Guide - Full configuration reference
• API Documentation - Complete API reference
• Architecture Guide - Overall system architecture