Satta Darshan
Civic political intelligence directory tracking Indian parliamentarians, ministers, and state assemblies with automated ingestion pipelines.
Executive Summary // 30-Second Recruiter Brief
Civic and parliamentary information in India is dispersed across government portals, unstructured tables, and fragmented records, making legislative tracking cumbersome for researchers and citizens.
Developed Satta Darshan — a full-stack Next.js and MongoDB platform with custom TypeScript/Node.js scrapers (Puppeteer, Cheerio) that fetch, validate, and normalize records of Lok Sabha & Rajya Sabha MPs from sansad.in, ministers, and state assemblies.
1. Problem Formulation & Motivation
Civic and parliamentary information in India is dispersed across government portals, unstructured tables, and fragmented records, making legislative tracking cumbersome for researchers and citizens.
To create an open-source civic data platform that aggregates, normalizes, and visualizes parliamentary records and politician dossiers in a clean, queryable interface.
2. Approach & System Solution
Developed Satta Darshan — a full-stack Next.js and MongoDB platform with custom TypeScript/Node.js scrapers (Puppeteer, Cheerio) that fetch, validate, and normalize records of Lok Sabha & Rajya Sabha MPs from sansad.in, ministers, and state assemblies.
3. Architecture & Execution Pipeline
Data engineering and web platform architecture combining automated Puppeteer scrapers, MongoDB normalized collections, and Next.js App Router with D3-Geo visualizations.
Satta Darshan — Topology
Interactive Visual Data Pipeline · Click any stage to inspect execution state
Official parliamentary directories (sansad.in) and public legislative sources.
Execution Data Flow Sequence
5 Steps- 1Automated TypeScript scraping scripts (scripts/sync-sansad.ts) launch headless Puppeteer browsers.
- 2Pipeline fetches member listings for Lok Sabha and Rajya Sabha from sansad.in with exponential retry logic.
- 3Data cleaning module normalizes names, state mappings, and constituency identifiers.
- 4Normalized records are upserted into MongoDB collections via Mongoose models.
- 5Next.js App Router renders politician profiles, parliamentary chamber rosters, and interactive geospatial state maps.
4. Engineering Decisions & Trade-Offs
Headless Puppeteer Scraping with Retry Logic
MongoDB Document Normalization for Dossiers
5. Implementation Snippet
import puppeteer from "puppeteer";
import connectDB from "../lib/db";
import Politician from "../models/Politician";
async function fetchWithRetry(page: any, url: string, retries = 3) {
for (let i = 0; i < retries; i++) {
try {
console.log(` Connecting to ${url} (Attempt ${i + 1})...`);
await page.goto(url, { waitUntil: "networkidle2", timeout: 300000 });
return true;
} catch (e: any) {
if (i === retries - 1) throw e;
await new Promise(r => setTimeout(r, 5000));
}
}
}
export async function syncSansadMembers(chamber: "Lok Sabha" | "Rajya Sabha") {
await connectDB();
const browser = await puppeteer.launch({ headless: true, args: ["--no-sandbox"] });
const page = await browser.newPage();
// Fetch and normalize parliamentary member records...
}6. Evaluation & Measured Results
Developed and published as an open-source civic intelligence platform featuring automated Puppeteer pipelines and structured legislative tracking.
Publicly available on GitHub (github.com/Rajchhapariya/SattaDarshan)
Parliamentary directory synchronized from official government portals
Full-stack platform with automated scraping and normalization pipelines
7. Limitations & Production Considerations
- •Scrapers require periodic maintenance if official government portal DOM structures change.
- •Historical term data completeness depends on public archive availability.