Lightweight and Scalable

Odilon is an Open Source Object Storage that runs on standard hardware.

It was designed as a redundant and secure file storage for applications that need to manage medium to large size objects (like pdfs, photos, audio, video).

It is small and easy to integrate, offers encryption, data protection and fault tolerance (software RAID and Erasure Codes) and detection of silent data degradation. Odilon also supports version control and master - standby replication over the Internet.

Main features

■ Scalable Object Storage on commodity disks
■ It has a simple single-level folder structure similar to the Bucket/Object model of Amazon S3
■ Single binary, does not need database or other external software
■ Runs on Linux and Windows
■ License Open Source Apache 2. It can be used for Open Source and commercial projects
■ HTTP/S for client server communication
■ Encryption at rest using modern algorithms (AES GCM-SIV)
■ Redundancy using Erasure Coding and software RAID. Tolerates full disk failures
■ Data immutability. Odilon supports two storage modes that protect data from deletion, whether accidental or intentional: Read Only and WORM (Write Once Read Many)
■ Master - Standby architecture with async replication over the web, for disaster recovery, high availability, archival, ransomware protection
■ Version Control
■ Simple operation. Adding new disks requires one line in the config file, and an async process sets up disks and replicata data in background
■ Disk monitoring for silent and slow data degradation detection (bit rot detection)
■ Integration with Key Management Server Hashicorp Vault
■ SDK Java 11 for client applications
■ Developed in Java (uses Spring Boot, OkHttp, Jackson, Caffeine, Dropwizard Metrics, among others)

Security

Odilon keeps objects encrypted (Encryption at Rest) using AES GCM-SIV.

Encryption has the following benefits:

■ It simplifies data protection, applications dont need to worry about protecting the data or encryption keys.
■ If data falls into the hands of an attacker, they cannot read it without also having access to the encryption keys. If attackers obtain the storage devices containing the data, they will not be able to understand or decrypt it.
■ It helps reduce the attack surface by removing lower layers of the hardware and software stack.
■ Simplify security management, centrally managed encryption keys create a single place where data access is enforced and can be audited.
■ It provides a privacy mechanism by limiting the access that systems and engineers have to data.

Each object stored by Odilon has a unique encryption key. Odilon uses envelope encryption (i.e. encrypting a key with another key), every object is encrypted with its unique key and the key is encrypted by Odilon key management layer or by a Key Management Server (KMS)

A KMS is software for generating, distributing, and managing cryptographic keys. It includes back-end functionality for key generation, distribution, and replacement. Moving key management to KMS prevents application reverse engineering attacks, simplifies operational maintenance, and compliance with security policies and regulations.

Odilon integrates with the KMS Open Source Hashicorp Vault.

Data Replication

Odilon can be configured to use software RAID for data replication. The supported configurations are

RAID 0. Two or more disks are combined to form a volume, which appears as a single virtual drive. It is not a configuration with data replication, its function is to provide greater storage and performance by allowing access to the disks in parallel.
RAID 1.For each object, 1 or more exact copies (or mirrors) are created on two or more disks. This provides redundancy in case of disk failure. At least 2 disks are required, Odilon also supports 3 or more for greater redundancy.
RAID 6 / Erasure Coding. It is a method of encoding data into blocks that can be distributed across multiple disks or nodes and then reconstructed from a subset of those blocks. It has great flexibility since you can adjust the number and size of the blocks and the minimum required for recovery. It uses less disk space than RAID 1 and can withstand multiple full disk failures. Odilon implements this architecture using Reed Solomon error-correction codes. The configurations are:
- 3 disks (2 data and 1 parity, supports 1 full disk failure)
- 6 disks (4 data and 2 parity, supports up to 2 full disks failure)
- 12 disks (8 data and 4 parity, supports up to 4 full disk failure)
- 24 disks (16 data and 8 parity, supports up to 8 full disk failure)
- 48 disks (32 data and 16 parity, supports up to 16 full disk failure)

Master Standby Architecture

Odilon supports Master - Standby Architecture for disaster recovery, high availability, archival, and ransomware protection. Data replication is done asynchronously using HTTP/S over the local network or the Internet. Setting up a standby server is simple, just add the URL and credentials to the master configuration file and restart. Odilon will propagate each operation to the standby server. It will also run a replication process in background for data existing before connecting the standby server.

What Odilon is not

Odilon is not a Distributed Storage like Cassandra, Hadoop etc.
Odilon supports master-standby architecture for archival, backup and data protection, but it is not a Distributed Storage and it does not support active-active replication.
Odilon is not a File System like GlusterFS, Ceph, ext4, etc.
It uses the underlying file system to stores objects as encrypted files, or in some configurations to break objects into chunks.
Odilon is not a NoSQL database like MongoDB, CouchDB, etc.
Odilon is not designed to store json or xml but files (like pdf or jpg). It does not use a database engine, it has its own journaling agent for Transaction Management and only supports simple queries, ie. to retrieve an object and to list the objects of a bucket filtered by objectname's prefix.
Odilon is not optimized for a very large number of small files
Odilon does not have optimization for lots of small files. The files are simply stored encrypted and compressed to local disks. Plus the extra meta file and shards for erasure coding.
Odilon API is not fully S3 compatible
Odilon API is simpler than S3. The only thing it has in common with AWS S3 it that uses the bucket/object methafor to organize the object space.

Using Odilon

A Java client program that interacts with the Odilon server must include the Odilon SDK jar in the classpath.
A typical architecture for a Web Application is

Example to upload 2 pdf files:


String endpoint = "http://localhost"; 

/** default port */
int port = 9234; 

/** default credentials */
String accessKey = "odilon";
String secretKey = "odilon";
										
String bucketName  = "demo_bucket";
String objectName1 = "demo_object1";
String objectName2 = "demo_object2";
										
File file1 = new File("test1.pdf");
File file2 = new File("test2.pdf");
										
/* put two objects in the bucket,
the bucket must exist before sending the object,
and object names must be unique for that bucket */
										
OdilonClient client = new ODClient(endpoint, port, accessKey, secretKey);

client.putObject(bucketName, objectName1, file1);
client.putObject(bucketName, objectName2, file2);

Download Odilon

To run Odilon Server you must have Java 17 installed.

The executable is the same for all platforms, the difference are the scripts to start and stop the program.

Odilon Client SDK Maven

From Maven Central

		
		
			<dependency>
				<groupId>io.odilon</groupId>
				<artifactId>odilon-client</artifactId>			
				<version>1.8</version>
			</dependency>

From Novamens' Maven Repository

		
		<repositories>
			<repository>
				<id>novamens-releases</id>
				<url>https://maven.novamens.com/nexus/content/repositories/releases/</url>
			</repository>
		</repositories>

		<dependencies>
			<dependency>
				<groupId>io.odilon</groupId>
				<artifactId>odilon-client</artifactId>			
				<version>1.8</version>
			</dependency>
		</dependencies>

Secure and redundant Open Source Object Storage

Lightweight and Scalable

Main features

Security

Data Replication

Master Standby Architecture

What Odilon is not

Using Odilon

Download Odilon

Odilon Server Linux

Odilon Server Windows

Odilon SDK client library

Odilon Client SDK Maven

Documentation

Installation and configuration

Java application development

Server design and development

Videos