# Bid Sentinel — Version 2

**UK Cyber Security Tender Platform** — Monitoring · Tracking · Analytics · Intelligence

A dockerized full-stack application that scrapes, tracks, and manages **UK public
and private sector tender opportunities** related to **Cyber Security Services** (MDR, SOC,
Incident Response, Vulnerability Management, CTI, and more).

| Layer        | Technology                                            |
|--------------|-------------------------------------------------------|
| Frontend     | React (Vite SPA) + Tailwind CSS, served by nginx      |
| Backend      | Python 3.12 + FastAPI (async)                         |
| Database     | PostgreSQL 16 (persistent Docker volume)              |
| Scraping     | HTTPX (official OCDS APIs) + BeautifulSoup4 fallback  |
| Auth         | JWT (python-jose) + bcrypt (passlib)                  |
| Orchestration| Docker & Docker Compose                               |

---

## 1. Architecture

```
┌────────────┐      ┌──────────────────────┐      ┌──────────────┐
│  Frontend  │ HTTP │       Backend        │ SQL  │  PostgreSQL  │
│ React/nginx│─────▶│  FastAPI (uvicorn)   │─────▶│   (volume)   │
│  :8080     │      │  :8000               │      │   :5432      │
└────────────┘      │  + background scraper│      └──────────────┘
                    └──────────┬───────────┘
                               │ HTTPX (async)
                ┌──────────────┴───────────────┐
                ▼                               ▼
       Find a Tender (FTS)            Contracts Finder
       OCDS Release API               OCDS Search API
```

- **Module A — Auth/Admin:** JWT login, bcrypt hashing, Admin vs Standard roles,
  30-day token for persistent sessions, admin-only user CRUD.
- **Module B — Scraper Engine:** async background scheduler scanning a registry
  of **11 UK bid portals** (toggleable), filtered by built-in **and** custom
  keywords, with URL/reference deduplication. Manual trigger via
  `POST /tenders/scrape`.
- **Module C — Schema:** see `backend/app/models.py` (`Tender` incl. `bid_status`
  and `outcome`, `User`, `Portal`, `CustomKeyword`, `ScheduleSlot`).
- **Module D — Dashboard:** a customizable **Performance overview** (KPI cards +
  charts with Count/Value, date-basis, range, and pivot toggles) above a table
  with search + status + **portal filter** and **sortable columns** (incl.
  **Fit %**), inline **bid-status** and **outcome** dropdowns (persist instantly),
  CSV + PDF export, and an admin **Settings** page.
- **Module E — Capability fit & certifications (deterministic, no AI):** upload a
  service-overview document (PDF/DOCX/TXT) to build a **capability profile**;
  each opportunity gets an explainable **Fit %** (share of your capability terms
  it mentions) plus the **certifications/accreditations** detected from a curated
  lexicon (Cyber Essentials Plus, ISO 27001, CHECK, SC/DV clearance, …). Fully
  offline — no API keys, no external calls. See `backend/app/analysis.py`.
- **Module F — Detail, collaboration & governance:** each row **expands** to show
  required certs/accreditations/**security clearances**, the **top technical
  requirements** scraped from the description, and a **comments** thread. Outcomes
  include a **Disregarded** option (hidden from the active view), and an
  admin-only **Audit trail** records logins and every change (who/what/when/IP).
- **Module H — Analyst Intelligence:** a read-only **Analyst** role and an
  **Intelligence** tab that mines the assessments for **capability gaps** (demand
  you don't cover), **certification/clearance gaps** (required vs an editable
  "accreditations held" list), **buyer intelligence** (target client
  organisations), **win/loss** patterns, **demand trends**, and **sales angles**
  (deterministic, or AI-written when the toggle is on) — with a printable
  **PDF Intelligence Report**. See `backend/app/intelligence.py`,
  `backend/app/reports.py`.
- **Module G — AI enhancement (optional, toggleable):** an admin switch in
  **Settings → AI enhancement**. When on (and `ANTHROPIC_API_KEY` is set), a
  **low-cost Claude model** (Haiku 4.5 by default — $1/$5 per 1M tokens) produces
  a richer summary, a reasoned fit score, and more accurate certs/clearances/
  technical requirements. When off or no key, Bid Sentinel stays fully
  deterministic and offline; every AI call degrades gracefully to the rules
  engine on error. See `backend/app/ai.py`.

### Bid Portals (Module B)

All portals ship with an **active scraper** and are **enabled by default**;
admins can untick any in **Settings → Bid Portals**, and **add their own** custom
portals there. Find a Tender and Contracts Finder use the official **OCDS JSON
APIs** (robust). The other nine use a **best-effort generic HTML scraper**
(`backend/app/scraper/generic_html.py`) — they are scraped on every run, but
because each site's HTML differs (and several require authenticated sessions),
their `search_url` / selectors may need per-site tuning to return results
reliably. See `backend/app/scraper/portals.py`.

| # | Portal | Scraper |
|---|--------|---------|
| 1 | Find a Tender Service (FTS) | **Live — OCDS API** |
| 2 | Contracts Finder | **Live — OCDS API** |
| 3 | Sell2Wales | Live — generic HTML (tune per site) |
| 4 | Public Contracts Scotland | Live — generic HTML (tune per site) |
| 5 | eTendersNI | Live — generic HTML (tune per site) |
| 6 | ProContract (Due North) | Live — generic HTML (auth/tune) |
| 7 | Delta eSourcing | Live — generic HTML (auth/tune) |
| 8 | In-Tend | Live — generic HTML (auth/tune) |
| 9 | Achilles | Live — generic HTML (auth/tune) |
| 10 | CCS / Government Commercial Agency | Live — generic HTML (tune per site) |
| 11 | Digital Marketplace (G-Cloud) | Live — generic HTML (tune per site) |

### Settings (admin)

- **Portals** — enable/disable each source.
- **Keywords** — built-in cyber terms (always on) + optional custom keywords.
- **Schedule** — run scrapes on **multiple days and times** (cartesian product of
  selected days × times), interpreted in `SCHEDULE_TIMEZONE` (default
  `Europe/London`). Falls back to `SCRAPER_INTERVAL_MINUTES` when no slots exist.

> **Note on scraping (API-first):** Bid Sentinel prefers a **JSON/OCDS API**
> wherever a portal publishes one — FTS and Contracts Finder use their official
> OCDS APIs, and any other portal can be pointed at an OCDS feed via
> `scraper/ocds_api.py`. Portals with no API use an **accurate HTML scraper**
> (`scraper/generic_html.py`) that parses structured result cards — title, buyer,
> value, and published/closing dates — rather than harvesting raw links. It's
> polite (one request per run), best-effort, and degrades gracefully (returns
> nothing rather than erroring) until tuned to each site's search page and given
> credentials where required. If every enabled portal returns nothing (e.g.
> offline), a small **sample dataset** is seeded so the dashboard is never empty.

---

## 2. Quick Start

### Prerequisites
- Docker Desktop (or Docker Engine + Compose v2)

### Steps

```bash
# 1. Copy and review environment variables
cp .env.example .env
#    (edit JWT_SECRET, ADMIN_PASSWORD, etc.)

# 2. Build and start everything
docker compose up --build -d

# 3. Watch backend logs (DB init, portal seed, first scrape)
docker compose logs -f backend
```

Then open:

| Service            | URL                          |
|--------------------|------------------------------|
| **Web Dashboard**  | http://localhost:8080        |
| Backend API docs   | http://localhost:8000/docs   |
| Health check       | http://localhost:8000/health |

### First-run admin setup
There is **no default account**. The first time you open the dashboard, the login
screen detects that no users exist and shows a **Create administrator account**
form — enter a name, email, and password (min 8 chars) and you're signed straight
in as admin. The setup endpoint (`POST /auth/setup`) locks itself once that first
account exists; subsequent visits show the normal sign-in form. Additional users
are created later by an admin via **Settings** / the `/users` API.

---

## 3. Database Initialization

No manual migration step is required. On startup the backend (see
`app/main.py` lifespan handler):

1. Connects to PostgreSQL (Compose waits for the DB healthcheck first).
2. Creates all tables via `Base.metadata.create_all`.
3. Seeds the portal registry (11 portals) if missing.
4. Starts the background scraper.

The admin account is **not** seeded — it is created via the first-run setup
screen (`POST /auth/setup`).

Data persists in the named Docker volume `pgdata`. To wipe everything and start
fresh:

```bash
docker compose down -v      # -v removes the pgdata volume
docker compose up --build -d
```

---

## 4. Common Commands

```bash
# Stop containers (keep data)
docker compose down

# Rebuild after code changes
docker compose up --build -d

# Trigger a scrape manually (admin token required) — easiest via the
# "Run Scraper" button in the UI, or:
curl -X POST http://localhost:8000/tenders/scrape \
     -H "Authorization: Bearer <token>"

# Open a psql shell
docker compose exec db psql -U ctt -d ctt -c "SELECT count(*) FROM tenders;"
```

---

## 5. API Overview

| Method | Path                       | Auth   | Description                       |
|--------|----------------------------|--------|----------------------------------|
| GET    | `/auth/setup-status`       | —      | Whether first-run setup is needed |
| POST   | `/auth/setup`              | —*     | Create first admin (locks after) |
| POST   | `/auth/login`              | —      | Obtain JWT (form: username/pass) |
| GET    | `/auth/me`                 | user   | Current user                     |
| GET    | `/tenders`                 | user   | List (bid_status, source, search, sort_by, order) |
| POST   | `/tenders`                 | user   | Manually add an opportunity      |
| PATCH  | `/tenders/{id}`            | user   | Edit an opportunity's details    |
| DELETE | `/tenders/{id}`            | user   | Delete an opportunity entirely   |
| POST   | `/tenders/bulk-delete`     | admin  | Delete selected + block from future scrapes |
| POST   | `/tenders/undo/{batch}`    | user   | Undo a deletion (restore + lift block) |
| PATCH  | `/tenders/{id}/status`     | user   | Update bid status (No Bid → outcome No Bid) |
| PATCH  | `/tenders/{id}/outcome`    | user   | Update outcome (Won/Lost/No Bid/Disregarded) |
| GET    | `/tenders/{id}/comments`   | user   | List comments                    |
| POST   | `/tenders/{id}/comments`   | user   | Add a comment                    |
| DELETE | `/tenders/{id}/comments/{cid}` | user* | Delete a comment (author/admin) |
| POST   | `/tenders/scrape`          | admin  | Run scrapers now                 |
| GET    | `/audit`                   | admin  | View the application audit trail |
| GET    | `/config/portals`          | user   | List portals + enabled state     |
| POST   | `/config/portals`          | admin  | Add a custom portal              |
| PATCH  | `/config/portals/{key}`    | admin  | Enable/disable a portal          |
| DELETE | `/config/portals/{key}`    | admin  | Delete a custom portal           |
| GET    | `/config/keywords`         | user   | Built-in + custom keywords        |
| POST   | `/config/keywords`         | admin  | Add a custom keyword             |
| DELETE | `/config/keywords/{id}`    | admin  | Remove a custom keyword          |
| GET    | `/config/cpv`              | user   | CPV codes (AND-narrow scrape)    |
| POST   | `/config/cpv`              | admin  | Add a CPV code + description      |
| PATCH  | `/config/cpv/{id}`         | admin  | Enable/disable a CPV code        |
| DELETE | `/config/cpv/{id}`         | admin  | Remove a CPV code                |
| GET    | `/config/cpv-strict`       | user   | CPV matching mode (lenient/strict) |
| PATCH  | `/config/cpv-strict`       | admin  | Set CPV mode (strict requires a match) |
| GET    | `/config/tuneout`          | user   | Tune-out (negative) terms        |
| POST   | `/config/tuneout`          | admin  | Tune out a type (+ purge non-cyber matches) |
| DELETE | `/config/tuneout/{id}`     | admin  | Remove a tune-out term           |
| GET    | `/config/schedule`         | user   | Slots + timezone + next run      |
| POST   | `/config/schedule`         | admin  | Add slots (days × times)         |
| PATCH  | `/config/schedule/{id}`    | admin  | Pause/resume a slot              |
| DELETE | `/config/schedule/{id}`    | admin  | Delete a slot                    |
| GET    | `/config/capabilities`     | user   | Capability profile terms         |
| POST   | `/config/capabilities`     | admin  | Add a capability term            |
| DELETE | `/config/capabilities/{id}`| admin  | Remove a capability term         |
| POST   | `/config/capabilities/upload` | admin | Parse doc → suggest terms       |
| GET    | `/config/certs`            | user   | Cert lexicon (informational)     |
| POST   | `/config/recompute`        | admin  | Re-run analysis (AI or rules)    |
| GET    | `/config/ai`               | user   | AI toggle state + availability   |
| PATCH  | `/config/ai`               | admin  | Enable/disable AI enhancement    |
| GET    | `/config/accreditations`   | user   | Accreditations you hold          |
| POST   | `/config/accreditations`   | admin  | Add an accreditation held        |
| DELETE | `/config/accreditations/{id}` | admin | Remove an accreditation held    |
| GET    | `/intelligence/summary`    | analyst | Headline intelligence KPIs      |
| GET    | `/intelligence/capability-gaps` | analyst | Demand you don't cover     |
| GET    | `/intelligence/cert-gaps`  | analyst | Required certs vs held          |
| GET    | `/intelligence/buyers`     | analyst | Buyer/client intelligence       |
| GET    | `/intelligence/winloss`    | analyst | Win/loss patterns               |
| GET    | `/intelligence/trends`     | analyst | Demand over time                |
| GET    | `/intelligence/angles`     | analyst | Sales angles (`use_ai` optional) |
| GET    | `/intelligence/report.pdf` | analyst | Printable Intelligence Report   |
| GET    | `/intelligence/management-summary.pdf` | analyst | Management pipeline summary PDF |
| GET    | `/email/status`            | user   | Whether outgoing email is set up |
| POST   | `/email/report`            | user   | Email a report to your own address |
| GET    | `/users`                   | admin  | List users                       |
| POST   | `/users`                   | admin  | Create user                      |
| PATCH  | `/users/{id}`              | admin  | Update user                      |
| DELETE | `/users/{id}`              | admin  | Delete user                      |

Interactive docs: **http://localhost:8000/docs**

---

## 6. Local Development (without Docker)

**Backend**
```bash
cd backend
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt
export DATABASE_URL=postgresql+asyncpg://ctt:ctt_password@localhost:5432/ctt
uvicorn app.main:app --reload
```

**Frontend**
```bash
cd frontend
npm install
VITE_API_BASE_URL=http://localhost:8000 npm run dev   # http://localhost:5173
```

---

## 7. Project Structure

```
cyber-tender-tracker/
├── docker-compose.yml
├── .env.example
├── README.md
├── make_user_guide.py         # regenerates docs/USER_GUIDE.pdf
├── docs/
│   ├── USER_GUIDE.pdf
│   ├── AI_API_KEY_SETUP.pdf   # how to set the optional AI key
│   ├── AI_API_KEY_SETUP.md    # (same guide, Markdown)
│   └── EMAIL_SETUP.md         # how to enable outgoing email
├── assets/
│   └── bid-sentinel-logo.png  # used by the PDF guide
├── backend/
│   ├── Dockerfile
│   ├── entrypoint.sh
│   ├── requirements.txt
│   └── app/
│       ├── main.py            # FastAPI app + lifespan (init/seed/scheduler)
│       ├── config.py          # env settings
│       ├── database.py        # async engine/session
│       ├── models.py          # User, Tender ORM
│       ├── schemas.py         # Pydantic models
│       ├── security.py        # bcrypt + JWT
│       ├── deps.py            # auth dependencies
│       ├── seed.py            # table creation + admin/portal bootstrap
│       ├── routers/           # auth, users, tenders, settings (config)
│       └── scraper/           # portals registry, scrapers, runner, scheduler
└── frontend/
    ├── Dockerfile
    ├── nginx.conf
    ├── package.json
    └── src/
        ├── api/client.js      # axios service layer
        ├── assets/bid-sentinel-logo.png
        ├── context/AuthContext.jsx
        ├── pages/Login.jsx
        ├── pages/Dashboard.jsx
        ├── pages/Settings.jsx # portals, keywords, schedule (admin)
        ├── components/AnalyticsPanel.jsx  # KPI cards + charts (Chart.js)
        ├── components/BidStatusSelect.jsx
        ├── components/OutcomeSelect.jsx
        └── utils/             # formatting + CSV/PDF export
```

---

## 8. Security Notes (Production)

- Change `JWT_SECRET` and `ADMIN_PASSWORD` before any non-local deployment.
- Remove the `db` port mapping (`5432:5432`) in `docker-compose.yml`.
- Put the backend behind TLS (reverse proxy) and tighten `CORS_ORIGINS`.
- Consider a refresh-token rotation scheme instead of a single long-lived token.
- Respect each portal's terms of use and rate limits when scraping.
