This article explains how Odilon encrypts objects on disk with the v2 record-based format. It is of potential interest to developers and administrators but is not necessary to use the product: encryption is transparent to client applications.

Odilon keeps objects encrypted (Encryption at Rest) with AES-GCM-SIV (RFC 8452). The v2 format encrypts an object as a sequence of small, independently authenticated records, so objects of any size are encrypted and decrypted using a small, fixed amount of memory.

Envelope encryption

Odilon uses envelope encryption (i.e. encrypting a key with another key):

• Every object is encrypted with its own random 256-bit key, the data encryption key (DEK).
• The DEK is encrypted (wrapped) once with the server master key, or by a Key Management Server if one is configured, and stored in the object header. The plaintext DEK is never written to disk.
• To read an object, Odilon reads the header, unwraps the DEK and decrypts the records one by one.

Stored object layout

An encrypted object is stored as a single stream: one header followed by a sequence of records.

[ header ]  magic | version | record size | wrap IV | wrapped DEK | base nonce
[ record 0 ]   length | ciphertext + 16-byte tag
[ record 1 ]   length | ciphertext + 16-byte tag
   ...
[ record N ]   length (final flag) | ciphertext + 16-byte tag

Each record carries up to 4 MiB of plaintext by default (encryption.recordSize in odilon.properties, valid range 4 KiB to 64 MiB). The record size is stored in each object’s header, so changing the setting only affects new objects: existing objects are always read with the record size they were written with.

How each record is protected

• Independent authentication
  Every record has its own AES-GCM-SIV authentication tag. Plaintext is released to the client only after the tag of its record has been verified.
• Unique nonce per record
  The nonce is the per-object random base nonce combined with the record index, so a nonce is never reused under the same key. AES-GCM-SIV also tolerates accidental nonce misuse.
• Ordering and position
  The record index is part of the authenticated data of each record. Reordered, duplicated or substituted records fail authentication.
• End of object
  The last record is flagged as final, and its authenticated data includes the total plaintext length. A truncated stream, a missing final record, or extra data after the end are detected and rejected. An empty object is a single empty final record.
• Fail closed
  Any authentication failure aborts the read with an error. Partially decrypted content is never treated as valid.

Bounded memory and large objects

AES-GCM-SIV computes its authentication tag over the whole message it processes, so an implementation must hold that message in memory. The older Odilon format applied the cipher to the entire object, which limited encrypted objects to about 2 GB and required memory proportional to object size.

The v2 format applies AES-GCM-SIV to one record at a time. Odilon never holds more than one record in memory, regardless of the object size, so there is no object size limit for encryption. Objects larger than 2 GB are now stored encrypted like any other object.

Storage layout: RAID 0, RAID 1 and Erasure Coding

The encryption layer works on byte streams and does not depend on how objects are stored:

• RAID 0
  the encrypted stream is stored as one file on one drive.
• RAID 1
  the encrypted stream is stored as one file on each mirror drive.
• Erasure Coding
  the encrypted stream is passed to the Erasure Coding encoder, which splits it into 32 MB stripes and encodes each stripe into Reed-Solomon shards. The encoder treats the ciphertext as plain bytes.

Encryption record boundaries are independent of Erasure Coding stripe boundaries: a record may span two stripes, and a stripe may contain several records. Objects that are not encrypted are stored as plain, unframed data.

Size overhead

The stored size of an encrypted object is the plaintext size plus a header of about 80 bytes plus 20 bytes per record (a 4-byte length and a 16-byte tag). With 4 MiB records the overhead is about 0.0005%: a 114 MB object takes about 114 MB plus 600 bytes.

Compatibility and upgrades

• Each object’s metadata records the encryption format used (encryptionFormat). Odilon chooses how to decrypt from the object’s own metadata, not from the current server configuration.
• Objects written by earlier versions remain readable and are not rewritten automatically. Objects without encryptionFormat are read with the original format.
• All new encrypted objects are written in the v2 format.
• An unknown format version is rejected with an error. Odilon never tries to decrypt an object with a different format.
• The object encryption process (which encrypts objects stored unencrypted) writes the v2 format. This includes objects that earlier versions had to leave unencrypted because of their size.
• Client applications are not affected: objects are decrypted by the server before being sent.

Source code

Encryption v2 Java source code. For how to enable encryption, see Encryption configuration; for the overall design, see Odilon Software Architecture.