News Pipeline is a robust, automated web service built with FastAPI that aggregates, enriches, and stores news articles from a variety of global sources. It acts as a centralized intelligence hub by fetching raw news from APIs (NewsAPI, The Guardian, Tiingo, Marketaux) and RSS feeds, processing them through NLP algorithms (TextBlob, RAKE) for sentiment analysis and keyword extraction, and seamlessly upserting the structured data into a MongoDB Atlas cluster. The project also features a built-in automated newsletter engine to send responsive HTML email digests to subscribers.

✨ Key Features

Multi-Source Aggregation: Fetches and normalizes news from diverse sources including NewsAPI, The Guardian, Tiingo, Marketaux, and custom RSS feeds.
Background Data Processing: Implements asynchronous background tasks to fetch, process, and store news without blocking HTTP requests, complete with progress-tracking endpoints.
NLP Enrichment Pipeline: Automatically detects article language, calculates sentiment polarity and subjectivity using TextBlob, and extracts relevant keywords using RAKE (Rapid Automatic Keyword Extraction).
Advanced Content Delivery: Exposes RESTful endpoints to retrieve articles with support for pagination, company/ticker filtering, and querying specifically for the newest batch of fetched data.
Newsletter & Subscription Engine: Built-in SMTP email system to generate responsive HTML news digests. Supports user subscriptions, unsubscriptions, individual on-demand emails, and mass broadcasts.
Command Line Interface (CLI): Provides a developer-friendly CLI to trigger data fetching, manage the subscriber list, and manually broadcast daily newsletters.
Robust Storage Architecture: Utilizes MongoDB Atlas for scalable document storage with intelligent upsert logic to prevent duplicate articles based on unique URL hashes.

🛠 Tech Stack

Backend Framework: Python, FastAPI, Uvicorn
Database: MongoDB (Atlas), PyMongo
NLP & Data Processing: TextBlob (Sentiment Analysis), RAKE (Keyword Extraction)
Email / Notification: SMTP, HTML Templates (Jinja2-style rendering)
Deployment & Containerization: Docker (Dockerfile included)