CloudVault: Building a Secure, Distributed and Intelligent Cloud Storage Platform

Modern cloud storage looks simple from the outside: upload a file, organize it into folders, share it, and download it later.

Behind that simple interface, however, there are much harder engineering problems:

How do you keep every user's files completely isolated?
How do you upload large files reliably over unstable networks?
What happens when an upload is interrupted halfway through?
How do file versions work across multiple storage providers?
How can public sharing remain secure without exposing the storage bucket?
Can semantic search and AI operate on private files without weakening privacy?
How do you test and observe a system that spans databases, object storage, serverless functions, queues, workers, and frontend state?

I built CloudVault to explore these problems as one coherent system.

CloudVault is a full-stack distributed cloud storage platform built around privacy, recoverability, large-file storage, secure sharing, intelligent retrieval, observability, and production-oriented engineering.

Live Application

https://cloudvault-distributed-cloud-storag.vercel.app/

GitHub Repository

https://github.com/adityamsr2606/CloudVault-Distributed-Cloud-Storage-Platform

What I Wanted CloudVault to Be

I did not want CloudVault to become another project where authentication is added to a file-upload form and the result is called "cloud storage."

The goal was to build something that explores the kinds of engineering boundaries found in real storage products.

CloudVault therefore includes:

private per-user storage
nested folders
direct and multipart uploads
file versioning
secure public sharing
file and folder Trash
provider-aware permanent deletion
realtime synchronization
semantic search
hybrid retrieval
optional grounded AI
MFA
Row Level Security
distributed backend infrastructure
CI/CD
responsive browser testing
metrics and tracing

A major design rule throughout the project is simple:

A user should never be able to discover or access another user's private data unless that data has been deliberately exposed through a controlled sharing mechanism.

Two Architectures, One Project

One of the more unusual parts of CloudVault is that it contains two runtime profiles.

They serve different engineering goals without replacing each other.

Hosted Product Architecture

The production application uses:

React + TypeScript + Vercel + Supabase + Backblaze B2

The hosted architecture separates responsibilities across several layers:

Vercel hosts the frontend.
Supabase Auth manages identity and sessions.
PostgreSQL stores metadata and application state.
Row Level Security protects user-owned rows.
Supabase Realtime synchronizes state changes.
Supabase Storage handles smaller private objects.
Backblaze B2 handles the large-object path.
Supabase Edge Functions perform trusted operations.
pgvector + PostgreSQL FTS power intelligent retrieval.
Gemini can optionally generate grounded answers when the user explicitly allows external AI.

Long-lived provider credentials never need to exist inside the browser.

Local Distributed-Systems Architecture

The repository also contains a deeper local infrastructure profile using:

FastAPI
PostgreSQL
Redis
RabbitMQ
Celery
MinIO
Elasticsearch
Prometheus
Grafana
OpenTelemetry
Docker Compose

This profile lets the project explore:

asynchronous workers
message queues
distributed service boundaries
local S3-compatible storage
caching
search infrastructure
metrics
tracing
failure isolation

The API and the background worker are intentionally separated so expensive or asynchronous work does not need to block normal API requests.

Designing for Private Per-User Storage

Authentication alone does not provide data isolation.

A frontend could hide another user's files while the database still permits access if requests are manually modified.

CloudVault therefore moves the ownership boundary deeper into the system.

PostgreSQL Row Level Security

Private tables use PostgreSQL Row Level Security based on the authenticated Supabase identity.

Conceptually:

using ((select auth.uid()) = owner_id)

This protects resources such as:

files
folders
versions
multipart sessions
document chunks
share links
activity records
AI events
user preferences

The important part is that the browser is not trusted as the only authorization layer.

Even if a request is manipulated, PostgreSQL still evaluates the ownership policy.

The Large-File Problem

Large uploads became one of the most interesting parts of the project.

CloudVault separates three different concepts:

Product upload ceiling
Direct-upload threshold
Large-object provider

The current runtime configuration is:

SettingValueProduct upload ceiling1.5 GiBDirect Supabase threshold50 MBLarge-object providerBackblaze B2Multipart part size16 MiBParallel part uploads3Retry attempts4

Files below the direct threshold can use private Supabase Storage.

Files above it are routed to Backblaze B2 through multipart upload.

Multipart Upload Architecture

Instead of sending a large file through one server request, CloudVault splits it into smaller parts.

For example:

64 MiB file

Part 1 -> 16 MiB
Part 2 -> 16 MiB
Part 3 -> 16 MiB
Part 4 -> 16 MiB

The browser requests temporary signed upload URLs from a trusted Edge Function.

It then uploads the parts directly to B2.

The flow looks roughly like this:

User selects a large file.
CloudVault creates a multipart session.
The file is divided into parts.
The browser requests presigned URLs.
Parts are uploaded directly to Backblaze B2.
B2 returns an ETag for each part.
CloudVault completes the multipart object.
PostgreSQL metadata is finalized.

This means the Edge Function controls authorization without acting as a large binary proxy.

Pause, Resume and Recovery

Large uploads are much more useful when the entire transfer does not have to restart after a temporary failure.

CloudVault's multipart design includes support for:

pause
resume
cancellation
retries
provider-side part recovery
multipart-session recovery
browser-refresh recovery

When an upload resumes, already completed parts do not need to be transferred again.

CloudVault can ask the storage provider which parts already exist and continue only with the missing ones.

That makes the system fundamentally different from a simple "retry the whole upload" implementation.

Why the Storage Provider Is Stored on the Upload Session

Runtime configuration can change while a large upload is in progress.

Suppose an upload begins using B2:

Upload starts
    |
Provider = B2
    |
Configuration changes later

That existing upload must still complete against B2.

For this reason, CloudVault records the provider on the multipart session itself instead of relying only on the current global configuration.

This prevents long-running uploads from drifting between providers.

Real Production Validation

I wanted the project documentation to distinguish between something that is implemented and something that has actually been verified in production.

The B2 path has been validated using a real browser upload of:

64 MiB

with:

4 multipart parts
16 MiB per part
authenticated multipart initiation
browser-to-B2 transfer
working CORS
returned ETags
multipart completion
final CloudVault metadata

This proves the application can successfully use the large-object path above the original direct-upload threshold.

The product is configured for files up to 1.5 GiB, but I intentionally do not describe the exact 1.5 GiB ceiling as production-verified until a real 1.5 GiB browser transfer succeeds end-to-end.

That distinction matters.

Secure File Sharing Without Public Buckets

CloudVault allows users to share individual files while keeping the object-storage bucket private.

A share link has its own policy:

expiration
maximum usage count
revocation
current usage count

The underlying storage file itself does not become public.

When a share link is created:

CloudVault generates a random token.
The token is hashed using SHA-256.
Only the hash is stored.
The raw token is returned in the public URL.

When someone opens the link:

CloudVault hashes the provided token.
It resolves the matching share record.
Expiration is checked.
Revocation is checked.
Usage limits are checked atomically.
The storage provider is resolved.
A short-lived signed object URL is generated.

The raw access token is therefore never stored in the database.

Provider-Aware File Versioning

File versioning gets more complicated once multiple object-storage providers exist.

A logical file might look like this:

report.pdf
|
+-- v1 -> Supabase Storage
|
+-- v2 -> Backblaze B2
|
+-- v3 -> Backblaze B2 (current)

Each version stores its own provider information.

This allows CloudVault to correctly:

download old versions
restore old versions
purge historical objects
survive provider changes

Restoring v1 does not overwrite the historical record.

Instead:

v1
v2
v3 current

Restore v1

v1
v2
v3
v4 current <- content restored from v1

This preserves immutable history.

Recoverable File and Folder Deletion

CloudVault separates Trash from permanent deletion.

Moving a file to Trash changes its logical lifecycle without immediately destroying the underlying object.

This allows:

restoration
safer destructive operations
delayed permanent purge

Folders are more complicated because they may contain nested folders and files.

CloudVault treats folder Trash as a hierarchy operation.

The system resolves descendants, groups the hierarchy under one Trash root, and restores the structure coherently.

Database RPCs are used for these multi-row operations so the hierarchy can be modified transactionally instead of through a chain of disconnected browser updates.

CloudVault Intelligence

CloudVault also contains an intelligent retrieval layer.

The goal was not to attach a generic chatbot to the application.

The goal was to make private files easier to discover and understand.

The retrieval pipeline combines:

document chunking
gte-small embeddings
pgvector similarity
PostgreSQL full-text search
hybrid ranking
owner-level authorization
file-level deduplication

A user query is evaluated through both semantic and lexical retrieval.

Conceptually:

semantic_weight × semantic_score
+
lexical_weight × lexical_score
=
hybrid relevance

Semantic retrieval helps when wording differs.

Lexical retrieval helps with exact:

names
identifiers
technical phrases
product names
terminology

Combining both produces a more balanced retrieval strategy.

Permission-Aware AI

Search and AI should not become a back door around storage authorization.

CloudVault therefore keeps retrieval owner-aware.

Private results are not retrieved globally and filtered only in React.

The ownership boundary remains in the data layer.

This applies to:

semantic search
hybrid search
related-file discovery
evidence retrieval
grounded answers

Grounded AI Instead of a Generic Chatbot

CloudVault can optionally use Gemini after private evidence has already been retrieved.

The external generation path is enabled only when:

the deployment allows generative AI
a valid server-side provider key exists
the user explicitly allows external AI

Without all three conditions, CloudVault remains in retrieval-only mode.

This is important because storing private files should not automatically imply permission to send those files to an external AI provider.

The generation layer is therefore optional, consent-gated, and downstream of retrieval.

Observability and Distributed Engineering

The local CloudVault profile includes:

Prometheus
Grafana
OpenTelemetry
structured logging
trace IDs
span IDs
request correlation

This provides visibility into more than whether an endpoint returned HTTP 200.

The architecture is designed to expose how the different services interact.

CI/CD and Quality Gates

CloudVault uses GitHub Actions as part of normal development.

The CI pipeline validates multiple layers.

Backend

Ruff linting
formatting
Pytest
Python compilation
Docker build

Frontend

dependency security audit
TypeScript checks
Vite production build
Playwright responsive QA
Docker build

Supabase Edge Functions

Deno typechecking

This means a successful code change has to survive several independent quality signals before it is treated as ready.

Responsive Browser Testing

The frontend is designed for:

desktop
tablet
mobile

Responsive behavior is checked automatically using Playwright.

The tests cover multiple viewport classes using Chromium and WebKit.

This helps catch layout regressions that simple TypeScript compilation cannot detect.

Frontend Performance Work

One measured optimization in CloudVault was route-level lazy loading.

The frontend originally loaded most route code eagerly.

After introducing React lazy and Suspense, the shared bundle changed from:

Build StateShared JavaScriptShared gzipBefore530.29 kB150.22 kBAfter445.34 kB130.04 kB

That is approximately:

16% less shared minified JavaScript
13% less shared gzip JavaScript

These are build-artifact measurements, not claims about browser latency or Core Web Vitals.

Technology Stack

CloudVault brings together several engineering domains.

Frontend

React 19
TypeScript
Vite
React Router
Tailwind CSS
Lucide React
Supabase JavaScript SDK
Playwright

Hosted Backend

Supabase Auth
PostgreSQL
Row Level Security
Realtime
Storage
Edge Functions

Object Storage

Supabase Storage
Backblaze B2
MinIO

Backend

Python 3.12
FastAPI
SQLAlchemy
Alembic
Psycopg
Uvicorn

Distributed Systems

Redis
RabbitMQ
Celery
Elasticsearch

Retrieval and AI

pgvector
PostgreSQL Full-Text Search
Supabase gte-small
Gemini

Observability

Prometheus
Grafana
OpenTelemetry

DevOps

Docker
Docker Compose
GitHub Actions
Vercel

What I Learned From Building CloudVault

The most interesting part of the project was not adding more frameworks.

It was understanding the boundaries between them.

Some of the most important engineering lessons were:

Authentication and authorization are different problems.
Object-storage metadata must remain provider-aware.
Large uploads need recovery state, not just progress bars.
Public sharing should not require public buckets.
File version history becomes harder when providers can change.
Recursive folder lifecycle operations belong in transactional workflows.
AI retrieval must preserve the same privacy boundary as normal file access.
Browser CORS can be a real part of distributed storage design.
CI cannot replace production validation.
Implemented architecture and verified production behavior should be documented separately.

Current Validation Status

Today, CloudVault has working engineering validation around:

private authenticated storage
owner-scoped RLS
nested folders
file versioning
Trash and restore
secure public sharing
Supabase Realtime
MFA
hybrid retrieval
pgvector
responsive QA
CI/CD
Backblaze B2 multipart uploads
real 64 MiB browser upload
B2-backed sharing
distributed local infrastructure
metrics and tracing

Areas still intentionally awaiting stronger production evidence include:

a real full-size 1.5 GiB upload
large-scale pause/resume testing
large-scale browser-refresh recovery
representative retrieval-quality benchmarks
meaningful production load testing
hosted passkey verification

Final Thoughts

CloudVault started as a cloud-storage project, but it gradually became an exercise in system boundaries.

Storage, databases, authentication, authorization, distributed workers, object providers, AI, observability, and frontend state all affect each other.

The project therefore follows one principle throughout its architecture:

A storage platform should not only store files. It should protect ownership, survive failures, preserve history, expose controlled access, remain observable, and make private information easier to retrieve without weakening its security model.

CloudVault

Private by design. Recoverable by architecture. Intelligent with explicit boundaries.

Secure Storage · Distributed Systems · Hybrid Retrieval · Production Engineering

Live:

https://cloudvault-distributed-cloud-storag.vercel.app/

GitHub:

https://github.com/adityamsr2606/CloudVault-Distributed-Cloud-Storage-Platform


### Images I would use in the article

You don't need generic stock photos. For this project, **actual product and engineering screenshots will look much stronger**:

1. **Hero image** — CloudVault landing/dashboard screenshot.
2. **Hosted architecture** — architecture diagram.
3. **Distributed architecture** — local services diagram.
4. **Security** — simple RLS/user-isolation diagram.
5. **Large uploads** — your real `64 MiB / 4 parts / Complete` screenshot.
6. **Secure sharing** — screenshot of the share dialog and public download page.
7. **Trash/versioning** — Vault or version history UI.
8. **CloudVault Intelligence** — intelligence/search interface.
9. **CI/CD** — successful GitHub Actions workflow.
10. **Observability** — Grafana/Prometheus screenshot if you have one.

For the final published article, I would strongly prefer **real CloudVault screenshots + your actual architecture diagrams** over AI-generated decorative images. That makes the article look like genuine engineering work rather than marketing material.