admin / Bid-Sentinel
publicBid Scrape and Tracking Application with AI Capability
Bid-Sentinel / bid-sentinel-v2 / README.md
18134 B · main
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 | # Bid Sentinel — Version 2 **UK Cyber Security Tender Platform** — Monitoring · Tracking · Analytics · Intelligence A dockerized full-stack application that scrapes, tracks, and manages **UK public and private sector tender opportunities** related to **Cyber Security Services** (MDR, SOC, Incident Response, Vulnerability Management, CTI, and more). | Layer | Technology | |--------------|-------------------------------------------------------| | Frontend | React (Vite SPA) + Tailwind CSS, served by nginx | | Backend | Python 3.12 + FastAPI (async) | | Database | PostgreSQL 16 (persistent Docker volume) | | Scraping | HTTPX (official OCDS APIs) + BeautifulSoup4 fallback | | Auth | JWT (python-jose) + bcrypt (passlib) | | Orchestration| Docker & Docker Compose | --- ## 1. Architecture ``` ┌────────────┐ ┌──────────────────────┐ ┌──────────────┐ │ Frontend │ HTTP │ Backend │ SQL │ PostgreSQL │ │ React/nginx│─────▶│ FastAPI (uvicorn) │─────▶│ (volume) │ │ :8080 │ │ :8000 │ │ :5432 │ └────────────┘ │ + background scraper│ └──────────────┘ └──────────┬───────────┘ │ HTTPX (async) ┌──────────────┴───────────────┐ ▼ ▼ Find a Tender (FTS) Contracts Finder OCDS Release API OCDS Search API ``` - **Module A — Auth/Admin:** JWT login, bcrypt hashing, Admin vs Standard roles, 30-day token for persistent sessions, admin-only user CRUD. - **Module B — Scraper Engine:** async background scheduler scanning a registry of **11 UK bid portals** (toggleable), filtered by built-in **and** custom keywords, with URL/reference deduplication. Manual trigger via `POST /tenders/scrape`. - **Module C — Schema:** see `backend/app/models.py` (`Tender` incl. `bid_status` and `outcome`, `User`, `Portal`, `CustomKeyword`, `ScheduleSlot`). - **Module D — Dashboard:** a customizable **Performance overview** (KPI cards + charts with Count/Value, date-basis, range, and pivot toggles) above a table with search + status + **portal filter** and **sortable columns** (incl. **Fit %**), inline **bid-status** and **outcome** dropdowns (persist instantly), CSV + PDF export, and an admin **Settings** page. - **Module E — Capability fit & certifications (deterministic, no AI):** upload a service-overview document (PDF/DOCX/TXT) to build a **capability profile**; each opportunity gets an explainable **Fit %** (share of your capability terms it mentions) plus the **certifications/accreditations** detected from a curated lexicon (Cyber Essentials Plus, ISO 27001, CHECK, SC/DV clearance, …). Fully offline — no API keys, no external calls. See `backend/app/analysis.py`. - **Module F — Detail, collaboration & governance:** each row **expands** to show required certs/accreditations/**security clearances**, the **top technical requirements** scraped from the description, and a **comments** thread. Outcomes include a **Disregarded** option (hidden from the active view), and an admin-only **Audit trail** records logins and every change (who/what/when/IP). - **Module H — Analyst Intelligence:** a read-only **Analyst** role and an **Intelligence** tab that mines the assessments for **capability gaps** (demand you don't cover), **certification/clearance gaps** (required vs an editable "accreditations held" list), **buyer intelligence** (target client organisations), **win/loss** patterns, **demand trends**, and **sales angles** (deterministic, or AI-written when the toggle is on) — with a printable **PDF Intelligence Report**. See `backend/app/intelligence.py`, `backend/app/reports.py`. - **Module G — AI enhancement (optional, toggleable):** an admin switch in **Settings → AI enhancement**. When on (and `ANTHROPIC_API_KEY` is set), a **low-cost Claude model** (Haiku 4.5 by default — $1/$5 per 1M tokens) produces a richer summary, a reasoned fit score, and more accurate certs/clearances/ technical requirements. When off or no key, Bid Sentinel stays fully deterministic and offline; every AI call degrades gracefully to the rules engine on error. See `backend/app/ai.py`. ### Bid Portals (Module B) All portals ship with an **active scraper** and are **enabled by default**; admins can untick any in **Settings → Bid Portals**, and **add their own** custom portals there. Find a Tender and Contracts Finder use the official **OCDS JSON APIs** (robust). The other nine use a **best-effort generic HTML scraper** (`backend/app/scraper/generic_html.py`) — they are scraped on every run, but because each site's HTML differs (and several require authenticated sessions), their `search_url` / selectors may need per-site tuning to return results reliably. See `backend/app/scraper/portals.py`. | # | Portal | Scraper | |---|--------|---------| | 1 | Find a Tender Service (FTS) | **Live — OCDS API** | | 2 | Contracts Finder | **Live — OCDS API** | | 3 | Sell2Wales | Live — generic HTML (tune per site) | | 4 | Public Contracts Scotland | Live — generic HTML (tune per site) | | 5 | eTendersNI | Live — generic HTML (tune per site) | | 6 | ProContract (Due North) | Live — generic HTML (auth/tune) | | 7 | Delta eSourcing | Live — generic HTML (auth/tune) | | 8 | In-Tend | Live — generic HTML (auth/tune) | | 9 | Achilles | Live — generic HTML (auth/tune) | | 10 | CCS / Government Commercial Agency | Live — generic HTML (tune per site) | | 11 | Digital Marketplace (G-Cloud) | Live — generic HTML (tune per site) | ### Settings (admin) - **Portals** — enable/disable each source. - **Keywords** — built-in cyber terms (always on) + optional custom keywords. - **Schedule** — run scrapes on **multiple days and times** (cartesian product of selected days × times), interpreted in `SCHEDULE_TIMEZONE` (default `Europe/London`). Falls back to `SCRAPER_INTERVAL_MINUTES` when no slots exist. > **Note on scraping (API-first):** Bid Sentinel prefers a **JSON/OCDS API** > wherever a portal publishes one — FTS and Contracts Finder use their official > OCDS APIs, and any other portal can be pointed at an OCDS feed via > `scraper/ocds_api.py`. Portals with no API use an **accurate HTML scraper** > (`scraper/generic_html.py`) that parses structured result cards — title, buyer, > value, and published/closing dates — rather than harvesting raw links. It's > polite (one request per run), best-effort, and degrades gracefully (returns > nothing rather than erroring) until tuned to each site's search page and given > credentials where required. If every enabled portal returns nothing (e.g. > offline), a small **sample dataset** is seeded so the dashboard is never empty. --- ## 2. Quick Start ### Prerequisites - Docker Desktop (or Docker Engine + Compose v2) ### Steps ```bash # 1. Copy and review environment variables cp .env.example .env # (edit JWT_SECRET, ADMIN_PASSWORD, etc.) # 2. Build and start everything docker compose up --build -d # 3. Watch backend logs (DB init, portal seed, first scrape) docker compose logs -f backend ``` Then open: | Service | URL | |--------------------|------------------------------| | **Web Dashboard** | http://localhost:8080 | | Backend API docs | http://localhost:8000/docs | | Health check | http://localhost:8000/health | ### First-run admin setup There is **no default account**. The first time you open the dashboard, the login screen detects that no users exist and shows a **Create administrator account** form — enter a name, email, and password (min 8 chars) and you're signed straight in as admin. The setup endpoint (`POST /auth/setup`) locks itself once that first account exists; subsequent visits show the normal sign-in form. Additional users are created later by an admin via **Settings** / the `/users` API. --- ## 3. Database Initialization No manual migration step is required. On startup the backend (see `app/main.py` lifespan handler): 1. Connects to PostgreSQL (Compose waits for the DB healthcheck first). 2. Creates all tables via `Base.metadata.create_all`. 3. Seeds the portal registry (11 portals) if missing. 4. Starts the background scraper. The admin account is **not** seeded — it is created via the first-run setup screen (`POST /auth/setup`). Data persists in the named Docker volume `pgdata`. To wipe everything and start fresh: ```bash docker compose down -v # -v removes the pgdata volume docker compose up --build -d ``` --- ## 4. Common Commands ```bash # Stop containers (keep data) docker compose down # Rebuild after code changes docker compose up --build -d # Trigger a scrape manually (admin token required) — easiest via the # "Run Scraper" button in the UI, or: curl -X POST http://localhost:8000/tenders/scrape \ -H "Authorization: Bearer <token>" # Open a psql shell docker compose exec db psql -U ctt -d ctt -c "SELECT count(*) FROM tenders;" ``` --- ## 5. API Overview | Method | Path | Auth | Description | |--------|----------------------------|--------|----------------------------------| | GET | `/auth/setup-status` | — | Whether first-run setup is needed | | POST | `/auth/setup` | —* | Create first admin (locks after) | | POST | `/auth/login` | — | Obtain JWT (form: username/pass) | | GET | `/auth/me` | user | Current user | | GET | `/tenders` | user | List (bid_status, source, search, sort_by, order) | | POST | `/tenders` | user | Manually add an opportunity | | PATCH | `/tenders/{id}` | user | Edit an opportunity's details | | DELETE | `/tenders/{id}` | user | Delete an opportunity entirely | | POST | `/tenders/bulk-delete` | admin | Delete selected + block from future scrapes | | POST | `/tenders/undo/{batch}` | user | Undo a deletion (restore + lift block) | | PATCH | `/tenders/{id}/status` | user | Update bid status (No Bid → outcome No Bid) | | PATCH | `/tenders/{id}/outcome` | user | Update outcome (Won/Lost/No Bid/Disregarded) | | GET | `/tenders/{id}/comments` | user | List comments | | POST | `/tenders/{id}/comments` | user | Add a comment | | DELETE | `/tenders/{id}/comments/{cid}` | user* | Delete a comment (author/admin) | | POST | `/tenders/scrape` | admin | Run scrapers now | | GET | `/audit` | admin | View the application audit trail | | GET | `/config/portals` | user | List portals + enabled state | | POST | `/config/portals` | admin | Add a custom portal | | PATCH | `/config/portals/{key}` | admin | Enable/disable a portal | | DELETE | `/config/portals/{key}` | admin | Delete a custom portal | | GET | `/config/keywords` | user | Built-in + custom keywords | | POST | `/config/keywords` | admin | Add a custom keyword | | DELETE | `/config/keywords/{id}` | admin | Remove a custom keyword | | GET | `/config/cpv` | user | CPV codes (AND-narrow scrape) | | POST | `/config/cpv` | admin | Add a CPV code + description | | PATCH | `/config/cpv/{id}` | admin | Enable/disable a CPV code | | DELETE | `/config/cpv/{id}` | admin | Remove a CPV code | | GET | `/config/cpv-strict` | user | CPV matching mode (lenient/strict) | | PATCH | `/config/cpv-strict` | admin | Set CPV mode (strict requires a match) | | GET | `/config/tuneout` | user | Tune-out (negative) terms | | POST | `/config/tuneout` | admin | Tune out a type (+ purge non-cyber matches) | | DELETE | `/config/tuneout/{id}` | admin | Remove a tune-out term | | GET | `/config/schedule` | user | Slots + timezone + next run | | POST | `/config/schedule` | admin | Add slots (days × times) | | PATCH | `/config/schedule/{id}` | admin | Pause/resume a slot | | DELETE | `/config/schedule/{id}` | admin | Delete a slot | | GET | `/config/capabilities` | user | Capability profile terms | | POST | `/config/capabilities` | admin | Add a capability term | | DELETE | `/config/capabilities/{id}`| admin | Remove a capability term | | POST | `/config/capabilities/upload` | admin | Parse doc → suggest terms | | GET | `/config/certs` | user | Cert lexicon (informational) | | POST | `/config/recompute` | admin | Re-run analysis (AI or rules) | | GET | `/config/ai` | user | AI toggle state + availability | | PATCH | `/config/ai` | admin | Enable/disable AI enhancement | | GET | `/config/accreditations` | user | Accreditations you hold | | POST | `/config/accreditations` | admin | Add an accreditation held | | DELETE | `/config/accreditations/{id}` | admin | Remove an accreditation held | | GET | `/intelligence/summary` | analyst | Headline intelligence KPIs | | GET | `/intelligence/capability-gaps` | analyst | Demand you don't cover | | GET | `/intelligence/cert-gaps` | analyst | Required certs vs held | | GET | `/intelligence/buyers` | analyst | Buyer/client intelligence | | GET | `/intelligence/winloss` | analyst | Win/loss patterns | | GET | `/intelligence/trends` | analyst | Demand over time | | GET | `/intelligence/angles` | analyst | Sales angles (`use_ai` optional) | | GET | `/intelligence/report.pdf` | analyst | Printable Intelligence Report | | GET | `/intelligence/management-summary.pdf` | analyst | Management pipeline summary PDF | | GET | `/email/status` | user | Whether outgoing email is set up | | POST | `/email/report` | user | Email a report to your own address | | GET | `/users` | admin | List users | | POST | `/users` | admin | Create user | | PATCH | `/users/{id}` | admin | Update user | | DELETE | `/users/{id}` | admin | Delete user | Interactive docs: **http://localhost:8000/docs** --- ## 6. Local Development (without Docker) **Backend** ```bash cd backend python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -r requirements.txt export DATABASE_URL=postgresql+asyncpg://ctt:ctt_password@localhost:5432/ctt uvicorn app.main:app --reload ``` **Frontend** ```bash cd frontend npm install VITE_API_BASE_URL=http://localhost:8000 npm run dev # http://localhost:5173 ``` --- ## 7. Project Structure ``` cyber-tender-tracker/ ├── docker-compose.yml ├── .env.example ├── README.md ├── make_user_guide.py # regenerates docs/USER_GUIDE.pdf ├── docs/ │ ├── USER_GUIDE.pdf │ ├── AI_API_KEY_SETUP.pdf # how to set the optional AI key │ ├── AI_API_KEY_SETUP.md # (same guide, Markdown) │ └── EMAIL_SETUP.md # how to enable outgoing email ├── assets/ │ └── bid-sentinel-logo.png # used by the PDF guide ├── backend/ │ ├── Dockerfile │ ├── entrypoint.sh │ ├── requirements.txt │ └── app/ │ ├── main.py # FastAPI app + lifespan (init/seed/scheduler) │ ├── config.py # env settings │ ├── database.py # async engine/session │ ├── models.py # User, Tender ORM │ ├── schemas.py # Pydantic models │ ├── security.py # bcrypt + JWT │ ├── deps.py # auth dependencies │ ├── seed.py # table creation + admin/portal bootstrap │ ├── routers/ # auth, users, tenders, settings (config) │ └── scraper/ # portals registry, scrapers, runner, scheduler └── frontend/ ├── Dockerfile ├── nginx.conf ├── package.json └── src/ ├── api/client.js # axios service layer ├── assets/bid-sentinel-logo.png ├── context/AuthContext.jsx ├── pages/Login.jsx ├── pages/Dashboard.jsx ├── pages/Settings.jsx # portals, keywords, schedule (admin) ├── components/AnalyticsPanel.jsx # KPI cards + charts (Chart.js) ├── components/BidStatusSelect.jsx ├── components/OutcomeSelect.jsx └── utils/ # formatting + CSV/PDF export ``` --- ## 8. Security Notes (Production) - Change `JWT_SECRET` and `ADMIN_PASSWORD` before any non-local deployment. - Remove the `db` port mapping (`5432:5432`) in `docker-compose.yml`. - Put the backend behind TLS (reverse proxy) and tighten `CORS_ORIGINS`. - Consider a refresh-token rotation scheme instead of a single long-lived token. - Respect each portal's terms of use and rate limits when scraping. |