Deduplication Engine - Documentation

📋 Overview

The Deduplication Engine is a modern, cloud-native solution for efficient data storage and backup. Built with Go and containerized with Docker, it provides enterprise-grade deduplication capabilities with microservices architecture.

99.92%

Deduplication Ratio

640

Chunks/Second

10MB/s

Processing Speed

4

Microservices

✨ Features

🔧 Variable-Block Chunking

Uses content-defined chunking with Blake3 hashing for optimal deduplication across different file types and sizes.

⚡ Intelligent Caching

LRU cache with Cuckoo filter for fast duplicate detection, reducing storage overhead and improving performance.

🏗️ Microservices Architecture

Distributed services for ingest, storage, and stream handling with gRPC communication for high performance.

🐳 Containerized

Full Docker support with docker-compose for easy deployment and scaling in any environment.

🗄️ Database Integration

CockroachDB for metadata storage with ACID compliance and distributed capabilities.

☁️ Object Storage

MinIO integration for scalable chunk storage with S3-compatible API.

🏗️ Architecture

Technology Stack

Language: Go 1.21+
Database: CockroachDB (PostgreSQL-compatible)
Object Storage: MinIO (S3-compatible)
Communication: gRPC with Protocol Buffers
Containerization: Docker & Docker Compose
Caching: LRU Cache with Cuckoo Filter
Chunking: Variable-block with Blake3 hashing

📊 Performance Results

File Type Testing

File Type	Size	Deduplication	Chunks	Result
Text (unique)	28B	0%	1	✅ Expected
Text (edited)	46B	0%	1	✅ Expected
Small binary	32B	0%	1	✅ Expected
Large binary	5MB	0%	640	✅ Expected
Compressed (.zip)	204B	0%	3	✅ Expected
Compressed (.gz)	60B	0%	1	✅ Expected
Repetitive	10MB	99.92%	1280	✅ Excellent

Performance Metrics

Chunking Speed: ~10MB/s
Deduplication Ratio: Up to 99.92% on repetitive content
Cache Hit Rate: >95% for duplicate detection
Storage Efficiency: Significant space savings on similar files

🚀 Quick Start

Prerequisites

Docker and Docker Compose
Go 1.21+ (for development)
Git

1. Clone the Repository

git clone https://github.com/radhakrish-venkat/dedupe-engine.git
cd dedupe-engine
            

2. Start the Services

docker-compose up -d

This will start:

CockroachDB: Database for metadata storage
MinIO: Object storage for chunks
Data Storage Node: gRPC server for storage operations
Ingest Node: gRPC server for backup processing

3. Test the System

# Test with a small file
docker run --rm --network dedupe-engine_dedupe-net \
  -v $(pwd):/data dedupe-engine-stream-handler \
  -file /data/test-file.txt -ingest-addr ingest-node:50051
            

4. Monitor Services

# Check service status
docker-compose ps

# View logs
docker-compose logs -f ingest-node
            

☸️ Kubernetes Deployment

Prerequisites

kind (Kubernetes IN Docker) cluster running
kubectl installed and configured
Docker (for building and loading images)
local-path provisioner (for dynamic PVC provisioning)

1. Build and Load Images

# Build all images (from project root)
docker build -f Dockerfile.data-storage -t dedupe-engine-data-storage-node:latest .
docker build -f Dockerfile.ingest -t dedupe-engine-ingest-node:latest .
docker build -f Dockerfile.stream-handler -t dedupe-engine-stream-handler:latest .
# Load images into kind cluster (replace 'desktop' with your cluster name if different)
kind load docker-image dedupe-engine-data-storage-node:latest --name desktop
kind load docker-image dedupe-engine-ingest-node:latest --name desktop
kind load docker-image dedupe-engine-stream-handler:latest --name desktop
            

2. Deploy to Kubernetes

cd k8s
./deploy-no-build.sh
            

3. Test the Deployment

kubectl apply -f stream-handler-job.yaml
# Check job status
kubectl get pods -n dedupe-engine
# View job logs
kubectl logs job/stream-handler-test -n dedupe-engine
            

Troubleshooting

PVCs Pending: Ensure the local-path provisioner is running: kubectl get pods -n local-path-storage
MinIO config errors: The deployment now uses explicit env vars; ensure minio-deployment.yaml is up to date.
Images not found: Always kind load docker-image ... after building images.

Cleanup

# Remove all resources and namespace
./cleanup.sh
            

🔍 API Reference

gRPC Services

BackupService (Ingest Node)

service BackupService {
  rpc StreamBackup(stream BackupRequest) returns (stream BackupResponse);
}
            

StorageService (Data Storage Node)

service StorageService {
  rpc StoreChunk(StoreChunkRequest) returns (StoreChunkResponse);
  rpc RetrieveChunk(RetrieveChunkRequest) returns (RetrieveChunkResponse);
}
            

Configuration

Environment Variables

Variable	Default	Description
COCKROACH_HOST	localhost	CockroachDB host
COCKROACH_PORT	26257	CockroachDB port
MINIO_ENDPOINT	localhost:9000	MinIO endpoint
MINIO_ACCESS_KEY	minioadmin	MinIO access key
MINIO_SECRET_KEY	minioadmin	MinIO secret key

📦 GitHub Repository

View the complete source code, contribute, and track issues on GitHub:

View on GitHub Report Issues Join Discussions

Repository Features

📁 Complete source code with Go modules
🐳 Docker configurations for all services
🧪 Comprehensive test suite
📚 Detailed documentation
🔄 CI/CD pipeline with GitHub Actions
📊 Performance benchmarks
🔧 Development tools and scripts