🚀 LLM Eval CI/CD Pipeline

Overview

An end-to-end evaluation platform for Retrieval-Augmented Generation (RAG) applications that automatically measures model quality whenever prompts, models, or retrieval configurations change.

Unlike traditional CI pipelines that only verify whether code builds successfully, this project introduces AI quality gates that evaluate metrics such as faithfulness, hallucination rate, abstention accuracy, latency, and throughput before allowing changes to pass.

✨ Key Features

Automated LLM evaluation pipeline
GitHub Actions integration for CI/CD
Quality-gate based merge validation
LLM-as-a-Judge evaluation
Real 100-case golden evaluation dataset
Interactive analytics dashboard
Historical run comparison
Trend visualization
Configurable RAG retrieval parameters
PostgreSQL-backed evaluation history

🏗 Architecture

GitHub Push
      │
      ▼
GitHub Actions
      │
      ▼
FastAPI Backend
      │
 ┌────┴────┐
 │Retriever│
 │ Ollama  │
 │ Judge   │
 └────┬────┘
      │
 PostgreSQL
      │
      ▼
React Dashboard

🛠 Tech Stack

Frontend

React
Vite
Tailwind CSS
Recharts

Backend

FastAPI
PostgreSQL (Neon)
SQLAlchemy
Ollama
Docker

DevOps

GitHub Actions
Vercel
Cloudflare Tunnel

📊 Evaluation Metrics

Faithfulness
Hallucination Rate
Abstention Accuracy
Average Latency
Tokens / Second
Pass / Fail Quality Gates

🎯 Why This Project?

Modern LLM applications frequently change prompts, retrieval strategies, and models, but many teams lack automated mechanisms to detect quality regressions.

This project brings software engineering best practices to AI systems by treating LLM evaluation as a CI/CD problem, automatically validating every change against a golden dataset before it is accepted.

🌐 Live Demo

Frontend

Deployed on Vercel.

Backend

The backend runs locally using FastAPI + Ollama and is exposed through a Cloudflare Tunnel during live demonstrations. Because Ollama performs local inference, pipeline execution is available only while the backend machine is running. Historical evaluation results remain viewable through the deployed dashboard.

🔮 Future Improvements

Cloud-hosted inference providers
Embedding-based retriever
Multiple judge models
Human evaluation integration
Persistent backend deployment