Odilon Architecture

This article is about the general architecture of the Odilon server. It is of potential interest to developers but is not necessary to use the product.

Odilon has three hierarchical layers (API, Object Storage, Virtual File System) and about a dozen general services used by them (Scheduler, Concurrency Control, Journal, Cache, Encryption, etc.).

API

This is the HTTP/S interface used by client applications, like Odilon Java SDK, it is a RESTFul API implemented with Spring Boot: Bucket and Object operations, System info, and others. They interact directly with the Object Storage, they do not see the lower layers. See API Controllers Java source code

Object Storage

It is essentially an intermediary that downloads the requirements into the Virtual File System. Object Storage Java source code

Virtual File System

The Virtual File System layer manages the repository on top of the OS File System. Odilon uses the underlying File System to store objects as encrypted files, or in some configurations to break objects into stripes, and each stripe into N shards. It implements software RAID ( RAID 0, RAID 1), and Erasure Coding designed to withstand full disk failures without data loss.

Journaled atomic operations

Odilon implements atomic object updates using write-ahead journaling and versioned backups. Each update is recorded in the journal before any changes are applied. The previous object state is preserved until the transaction commits, allowing automatic rollback after a failure. After a successful commit, the backup is discarded or retained as an earlier version if version control is enabled.

Erasure Coding

It is a method of encoding data into blocks (called shards) that can be distributed across multiple disks or nodes and then reconstructed from a subset of those blocks. It has great flexibility since you can adjust the number and size of the blocks and the minimum required for recovery. It uses less disk space than RAID 1 and can withstand multiple full disk failures. Odilon implements this architecture using Reed Solomon error-correction codes.

Volumes

Odilon organizes Erasure Coding storage into volumes. Each volume consists of N data directories (N=3, 6, 12, 24, 48). In a typical deployment, the server starts with a single volume containing N directories.

Encoding

Odilon splits objects into 32 MB stripes, and each stripe is then converted into N Reed Solomon shards (which include redundancy).

If encryption is enabled, the objects are encrypted and then split and encoded with Reed-Solomon. This whole process is performed on the byte stream in one pass.

Full disk failures

Erasure Coding protects against complete disk failures without data loss. The number of disks that can fail depends on the size of the volume. For example, a 3-disk volume can tolerate one disk failure, while a 6-disk volume can tolerate two.

After a failed disk is replaced and the server is restarted, Odilon automatically rebuilds the lost data in the background. The rebuild process is asynchronous, so the server is fully operational immediately and does not need to wait for it to complete.

Erasure Coding. Adding capacity

The server can be expanded by adding new storage in groups of N data directories, where N = 3, 6, 12, 24, or 48.

Odilon organizes Erasure Coding storage into volumes. Each volume consists of N data directories. In a typical deployment, the server starts with a single volume containing N directories.

When additional capacity is required, a new volume can be added: the administrator declares them as a new volume (dataStorage.volume. and the volume number, which is also the the volume id) and sets volume.active to the new volume id in odilon.properties. The new volume must also contain N data directories. Once added, it becomes the active volume, while all previously existing volumes are automatically switched to read-only mode.

Only one volume can be active at any given time. All new objects are written to the active volume, while existing objects remain accessible in the older read-only volumes.

Data scrubber

Odilon runs a background data scrubber that periodically walks every object in storage and verifies its integrity. The process runs in two phases:

Phase 1 — Detection. For each object the scrubber decodes the head version and compares the SHA-256 hash of the decoded payload against the hash stored in the object’s metadata. If the hashes match, the object is intact and no further action is taken.

Phase 2 — Shard identification and repair. If a mismatch is detected, the scrubber verifies the SHA-256 checksum of every individual shard file to identify the exact positions of the corrupt shards. Those positions are then treated as erasures — locations whose index is known — before the Reed-Solomon decoder is invoked. Converting errors of unknown position into erasures of known position restores the full parity capacity: up to P simultaneous corrupt shards can be recovered for a volume with P parity shards, the same tolerance as a full-disk failure. The payload is then re-encoded with fresh Reed-Solomon shards and correct parity, and the metadata (per-shard hashes, etag, integrity timestamp) is updated atomically via the journal.

This approach complements read-repair: while read-repair only triggers when an object is actively requested, the scrubber detects and repairs silent corruption in objects that are rarely or never accessed, preventing gradual data degradation from going unnoticed.

The scrubber is enabled by default and runs on a configurable schedule.

Read-repair strategy

Odilon can detect and repair silent data corruption caused by bad sectors, bit flips, or other storage errors. At write time, the SHA-256 checksum of each individual Reed-Solomon shard is computed and stored in the object's metadata (sha256Blocks). During every subsequent read, each shard is hashed in memory and compared against its stored checksum. Any shard whose digest does not match is treated as an erasure and reconstructed from the remaining healthy shards in a single RS decode pass. The repaired shard is then written back to disk so future reads find it intact. This process is O(N) SHA-256 hashes over data already loaded in memory and adds no extra disk I/O.

Because this verification runs on every read, it adds a small CPU cost proportional to the total shard data size. On filesystems without built-in block checksums — such as ext4 or XFS — this is the only application-level protection against silent corruption and should be enabled. On filesystems that already provide end-to-end integrity guarantees — such as ZFS (with checksum=sha256 or checksum=fletcher4) or Btrfs — the filesystem catches corruption before Odilon sees the bytes, making the shard-level SHA-256 pass redundant. In those environments it can be disabled by setting ec.shardChecksumVerify=false in odilon.properties to eliminate the unnecessary CPU overhead.

Scheduler

The Scheduler is a persistent job queue processor. It can manage multiple job queues, each with a specific execution policy (normally FIFO: First In, First Out) and semantics on job failure (ie. retry N times and continue, block and retry until the job can complete successfully, ...). The Scheduler persists jobs on the Virtual File System.

A Scheduler Worker is the main component of the Scheduler. It is a job queue with its' own policies and Thread pool to process the queue. It creates a dependency graph with the jobs that are to be executed in parallel in each batch in order to warrant that after the execution of the batch the end result will be equivalent to a sequential execution. The main scheduler workers are:

  • Standard

    CRUD operations after the Transaction is committed by the Journal Service.
  • CronJob

    Cron jobs that execute regularly based on a Cron expression, they are non blocking.
  • Standby replication

    Executes asynchronous replication to the standby Odilon server. This worker will not be started if there is no Standby server connected. The semantics of the replica queue is strict order, If a job can not be completed the queue will block until it can be completed.

Journal

The default method by which Odilon implements atomic commit and rollback is a Rollback Journal. When a thread wants to execute an operation, it first records the original unchanged content in a rollback journal(Java source code).

Encryption

Odilon keeps objects encrypted (Encryption at Rest) using modern algorithms such as AES GCM-SIV.

Odilon uses envelope encryption (i.e. encrypting a key with another key), every object is encrypted with its unique key and the key is encrypted by Odilon key management layer or by a (Key Management Server). (Java source code).

Concurrency Control

Odilon uses a lock-based resource protection on Read and Write provided by the Lock service (Java source code).

Replication

Service that asynchronously propagates operations already completed (after commit) to the Standby server (Java source code).

Other Services

Authentication, ObjectMetadata Cache and File Cache (both use the Caffeine library), Traffic Control to prevent resources overflow, Health Check and Monitoring (uses the Dropwizard Metrics library), Settings among others.