admin / Syanpse-Vanguard

# Synapse-Vanguard `v0.3.0`

## What's new in v0.3.0
- **Ingress API for external Synapse apps (IronNode).** An admin generates an
  API key (Admin ▾ → *Ingress API (IronNode)*); the key is vault-encrypted and
  persisted, shown with the ingest URL for copy-out, toggled on/off, and
  regenerated on demand (regenerating invalidates the old key immediately).
- **`POST /v1/ingest/events`** — key-authenticated (`Authorization: Bearer <key>`)
  push endpoint. Body `{"events":[…]}`; each event flows through the normal
  broker → normalizer → storage path via an `ironnode` parser that preserves
  severity / category / MITRE. An empty batch doubles as a connection probe.
- Sender side lives in **Synapse IronNode** (`settings` → *Synapse-Vanguard SIEM
  forwarder*): paste the URL + key, Test Connection, incidents forward on trigger.

---

# Synapse-Vanguard `v0.2.0`

Self-hosted, data-sovereign SIEM — part of the Synapse suite (IronNode design
system). This repository currently implements **Milestones 1–3 + deployable
agents**:

## What's new in v0.2.0
- **Expanded Windows telemetry** — agents now collect PowerShell, Defender, Task
  Scheduler, WMI-Activity and TerminalServices channels in addition to
  Security/System/Application/Sysmon.
- **Event enrichment** — the winlog parser maps high-signal Event IDs (failed
  logons, log clears, service/task installs, Sysmon, PowerShell script blocks,
  Defender detections) to a security-weighted **severity**, **category**, and
  **MITRE ATT&CK technique**.
- **auditd high-signal ruleset** — `agents/linux/audit.rules` is installed and
  loaded by the Linux installer, turning `/var/log/audit/audit.log` into a rich
  syscall-level source (opt out with `--no-audit`).
- **Colour-coded severity tags** — each severity has a distinct colour; critical
  tiers render solid + glow so they stand out. ATT&CK technique chips appear
  inline on matching events.
- **New logo** on the login screen and dashboard header (drop your PNG at
  `sv/web/assets/logo.png`; an IronNode SVG placeholder ships as fallback).
- **Agent source IP** shown on each agent card (captured at ingest/enrolment,
  honours `X-Forwarded-For`; auto-migrates the `last_ip` column onto older DBs).
- **Events-per-second meter** in the toolbar (`GET /v1/metrics`, 5s average).
- **Admin menu** (top-right, TENANT_ADMIN only) with four panels:
  User management (create / enable-disable / role / password reset, with
  last-admin & self-lockout guards), Agent management (enrollment token + revoke),
  Ingestion / data sources (EPS, events-by-source, agent channels), and System
  settings (version, backend/broker, secret-configured flags, counts — no
  secrets are ever returned).
- **Reused-volume continuity** — the Compose project name is pinned to
  `synapse-vanguard`, so this stack adopts v1's data volume and already-enrolled
  agents keep shipping without re-enrolment.


| # | Milestone | Status |
|---|-----------|--------|
| 1 | Pluggable storage engine (DuckDB embedded, ClickHouse-ready abstraction) + schema | ✅ |
| 2 | Agent ingest (mTLS / dev-bearer) → Redis Streams broker → normalizer workers | ✅ |
| 3 | Query API + RBAC (3 roles) + JWT auth + tenant isolation | ✅ |
| — | Linux & Windows deployable agents (disk-buffered, zstd, reliable replay) | ✅ |

Later milestones (agentless listeners, detections, AI module, MFA/federation,
CAF reporting) are scaffolded at the interface level but not built here.

## Architecture at a glance

```
agents (Win/Linux) --mTLS+zstd--> agent-ingest --> Redis Streams --> normalizer --> DuckDB
                                                                                       ^
                                          UI / analysts --> query API (RBAC+JWT) ------/
```

- **Storage Abstraction Layer** (`sv/storage/base.py`): the whole app depends on
  the `StorageEngine` interface. `SV_STORAGE_BACKEND=duckdb|clickhouse` selects the
  driver at startup — no caller changes to scale out.
- **Broker** (`sv/broker.py`): Redis Streams in prod (`SV_BROKER=redis`),
  in-process queue for single-process dev/tests (`SV_BROKER=memory`).
- **RBAC** (`sv/auth/rbac.py`): `READ_ONLY_VIEWER` / `SECURITY_ANALYST` /
  `TENANT_ADMIN`, mapped 1:1 to Synapse Sonar so identities are portable.
- **Vault** (`sv/vault.py`): AES-256-GCM, ported from Sonar's `lib/crypto.ts`.

## Quick start (no Docker, no Redis — single process)

```bash
pip install -r requirements.txt
PYTHONPATH=. python scripts/smoke.py          # end-to-end self-test, all green
```

## Deployment model (important)

Embedded **DuckDB is single-writer** — only one OS process may hold the DB file.
So the default deployment is a **single consolidated process** (`sv.main:app`)
that serves the API, web UI, and agent-ingest routes and runs the normalizer as
a background task, all on one shared engine. Running the pieces as separate
containers against the same DuckDB file causes a lock-conflict crash loop.

Fanning out into separate microservice containers only becomes valid with an
external shared DB (`SV_STORAGE_BACKEND=clickhouse`) — a later milestone.

## Web UI

A self-contained IronNode-themed dashboard (login + live event search + agent
status), served at the app root:

- **http://localhost:5500/** — the dashboard
- http://localhost:5500/docs — Swagger API explorer

Single `sv/web/index.html`, hand-rolled CSS using the Synapse IronNode tokens —
**no CDN, no build step, no external asset fetches** (air-gap friendly). Log in
with the bootstrap admin from your `.env`.

> Note: the "min severity" filter is applied client-side over the returned page;
> server-side severity-range filtering is a small `sv/query/builder.py` follow-up.

## Run with Docker Compose (recommended)

`.env` is auto-loaded by Compose. Generate one with real secrets, then bring it
up (one app container + Redis):

```bash
cp .env.example .env        # or let the provided .env stand; set strong secrets
docker compose up -d --build
# open http://localhost:5500/
```

## Run locally without Docker (single process, no Redis needed)

```bash
pip install -r requirements.txt
export $(grep -v '^#' .env | xargs)      # or set SV_* vars yourself
SV_BROKER=memory PYTHONPATH=. uvicorn sv.main:app --port 5500
# open http://localhost:5500/
```

Log in and search from the CLI:

```bash
TOKEN=$(curl -s localhost:5500/v1/auth/login \
  -H 'content-type: application/json' \
  -d '{"username":"admin","password":"<SV_BOOTSTRAP_ADMIN_PASSWORD>"}' | jq -r .access_token)

curl -s localhost:5500/v1/search -H "authorization: Bearer $TOKEN" \
  -H 'content-type: application/json' \
  -d '{"equals":{"source_type":"winlog"},"limit":50}' | jq
```

## Self-test

```bash
PYTHONPATH=. python scripts/smoke.py          # 20 checks, all green
```

## Optional: mTLS-hardened ingest (production)

To terminate the agent-ingest port with mutual TLS on its own port, generate
certs (`./certs/gen-dev-certs.sh`), set `SV_MTLS_ENABLED=true`, and run a second
process bound to the ingest port with TLS:

```bash
uvicorn sv.ingest.agent_api:app --port 9443 \
  --ssl-keyfile certs/server.key --ssl-certfile certs/server.crt \
  --ssl-ca-certs certs/ca.crt --ssl-cert-reqs 2
```

(Point agents at that port and give them client certs — see the agent config.)

## Deploy an agent

### Linux — one-shot installer (recommended)

Copy the `agents/` directory to the host, then:

```bash
sudo agents/linux/install-linux-agent.sh \
  --server <SERVER_IP> --token <SV_ENROLLMENT_TOKEN>
```

It creates an isolated venv, writes `/etc/sv-agent/agent.config.toml`, and
installs + starts a `sv-agent` systemd service (auto-detects Debian vs RHEL log
paths). Re-run to update in place; `--uninstall` to remove. `--help` for options.

### Manual (any platform)

```bash
export SV_AGENT_CONFIG=/etc/sv-agent/agent.config.toml   # see agents/agent.config.example.toml

# Linux
PYTHONPATH=. python -m agents.linux.sv_agent_linux

# Windows (PowerShell)
$env:SV_AGENT_CONFIG="C:\ProgramData\sv-agent\agent.config.toml"
python -m agents.windows.sv_agent_windows
```

Build standalone executables with PyInstaller — see the module docstrings in
`agents/linux/` and `agents/windows/`.

## Tests

```bash
PYTHONPATH=. python scripts/smoke.py           # 20 checks: storage/ingest/normalize/query/rbac
```

## What's deliberately stubbed

- `sv/storage/clickhouse_engine.py` — interface present, bodies raise
  `NotImplementedError` (scale-out milestone).
- mTLS CSR **signing** — the enrollment endpoint binds the agent's cert CN and
  returns the CA chain; wiring the server-CA to sign `csr_pem` lives in ops.
  Dev mode (`SV_MTLS_ENABLED=false`) is fully working via bearer ingest tokens.
- Windows native `win32evtlog` subscription is sketched; the exercised path is
  the zero-dependency PowerShell `Get-WinEvent` poller.