DatasetRadar — Product Demo

What it is

• I watch a set of news/journal websites and scrape new articles as they publish.
• I run each article through an LLM detection pass that identifies which datasets, surveys, or reports it mentions (e.g. "NFHS-5", "National Family Health Survey") — no manual tagging.
• I ground extracted names against a dataset registry by embedding similarity, so paraphrases resolve to the same canonical dataset instead of creating duplicates.
• I confidence-gate every detection: high-confidence hits auto-accept, medium-confidence hits wait in a human Review Queue, low-confidence hits get discarded.
• I've built it to produce a live, queryable index of who is citing which dataset, where, and how often — plus a natural-language Ask interface with cited answers.

Tech Stack

• Frontend: React 18 + React Router + TanStack Query, Vite, Tailwind CSS
• Backend API: FastAPI, SQLAlchemy 2.0, Alembic migrations, Pydantic v2
• Database: PostgreSQL 16 + pgvector (article & dataset embeddings)
• Task queue: Celery workers + Celery Beat, backed by Redis
• Scraper: httpx + BeautifulSoup + lxml, Playwright for JS pages, tenacity retries
• Dataset detection: LLM extraction (Ollama via LangChain), grounded against the registry by embedding similarity
• RAG Q&A: Natural-language question → embedding search over articles → cited answer
• Deployment: Docker Compose — dev stack (live reload) + prod stack (Nginx, resource limits, internal-only db/redis)

Core Workflow

Trending Datasets

This is the default landing page I built. It ranks every auto-discovered dataset by how many articles mention it, with first/last-seen dates and a per-domain filter (health, education, politics...) — the "what's hot right now" view.

Summary — Dataset Usage by Website

I built this as a cross-tab of dataset × website, restricted to accepted mentions above the 85% confidence bar, and made it exportable to CSV for reporting.

Review Queue — Human in the Loop

I route every medium-confidence detection (75–85%) here instead of auto-accepting it. A reviewer can accept, reject, or ask the LLM for a second opinion before it counts as confirmed.

Scrape Logs — Operational Visibility

I built per-website health cards (last run, total runs, success/partial/failure) plus a full run history: duration, articles found, new articles, datasets found, and errors.

URL Manager — Configure What Gets Watched

From here I can add a new website with a scrape frequency, trigger an ad-hoc scrape, or disable a source. Six sources are tracked in this demo instance.

Dataset Registry — The Canonical Dataset List

This is the set of "known" datasets I maintain. New datasets also arrive bottom-up: my LLM auto-creates registry entries for datasets it finds that aren't registered yet.

Ask — Natural-Language, Cited Q&A Over the Corpus

I built this so you can type a question in plain English; my RAG pipeline embeds it, retrieves the nearest matching scraped articles, and has the LLM synthesize a grounded, cited answer. (Captured mid-query — I run local Ollama inference on commodity CPU.)