feat(insights): ship deduplicated log bundles to nethesis-insights

.github/workflows/test-unit.yml

@@ -0,0 +1,14 @@
+name: Unit tests
+
+on:
+  push:
+    branches: ["**"]
+  pull_request:
+
+jobs:
+  pytest:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Run unit tests
+        run: ./test-unit.sh

.gitignore

@@ -0,0 +1,9 @@
+
+# local scratch: OpenRouter credentials, not for commit
+open_router
+test.sh
+
+__pycache__/
+*.pyc
+
+tests/outputs/

README.md

@@ -85,12 +85,125 @@ api-cli run module/loki1/get-configuration
 {
   "retention_days": 7,
   "active_from": "2021-05-28T15:49:27Z+00:00",
-  "active_to": "2021-05-28T15:49:27Z+00:00"
+  "active_to": "2021-05-28T15:49:27Z+00:00",
+  "insights": {
+    "status": "active",
+    "base_url": "https://insights.nethesis.it",
+    "verify_tls": true,
+    "subscription_configured": true,
+    "last_run": "Wed 2026-08-07 14:00:11 UTC"
+  }
 }
 ```
 
 Note: `active_to` field WILL miss if the instance is still active.
 
+### `set-insights`
+
+Configure the insights collector. On its 15-minute timer it collects a window
+of the cluster journal, scrubs likely secrets, masks variable text,
+deduplicates the result into counted templates, and ships the bundle to the
+Nethesis insights service, where the actual (LLM-based) analysis happens.
+The node performs no analysis and holds no LLM credential. Disabled by
+default.
+
+#### Parameters
+
+- `active`: enable or disable `insights-collector.timer`. Required.
+- `base_url`: base URL of the insights server. Required when `active` is
+  `true`.
+- `verify_tls`: verify the server TLS certificate. Optional, default `true`.
+  Set to `false` only for a self-signed test server — never against a
+  production endpoint.
+
+No API key is required or accepted any more. Identity comes from the node's
+existing NethServer subscription: the collector reads `system_id` and its
+secret from the `cluster/subscription` Redis hash at run time and
+authenticates as `Authorization: Basic base64(system_id:secret)`. A node with
+no subscription ships nothing and says so in the journal.
+
+#### Example
+
+```bash
+api-cli run module/loki1/set-insights --data '{
+  "active": true,
+  "base_url": "https://insights.nethesis.it"
+}'
+```
+
+Disable it again:
+
+```bash
+api-cli run module/loki1/set-insights --data '{"active": false}'
+```
+
+#### Findings
+
+Findings are no longer written to the local journal: analysis happens on the
+insights server, and findings are read back through its API, not through this
+module. The node's only journal output is operational, one line per window
+under `SYSLOG_IDENTIFIER=loki1/insights-collector`: a
+`shipped N templates, M lines -> 202` line on success, an error line
+otherwise.
+
+Check the collector's own health with:
+
+```bash
+runagent -m loki1 journalctl --user -u insights-collector
+```
+
+#### Manual execution
+
+The collector is also a plain CLI with three flags, so systemd invokes it
+with none:
+
+```bash
+# See exactly what would leave the node before enabling anything.
+# No subscription needed, no server URL needed, nothing is shipped.
+runagent -m loki1 ../bin/insights-collector --print
+```
+
+`runagent` changes directory to the module state directory, hence the
+`../bin/` prefix.
+
+| Flag | Effect |
+|------|--------|
+| `--print` | build the bundle and write it to stdout instead of shipping; needs no subscription and no server URL |
+| `--max-lines N` | cap on log lines read per window, before deduplication. Default `500` |
+| `--minutes N` | window size in minutes. Default `15` |
+
+A run covers one window and exits. A failure is loud and costs exactly one
+window: the next timer fire retries.
+
+#### Sizing
+
+What ships is deduplicated *templates*, not raw log lines, so the outbound
+volume is far below the raw line count of a window. `--max-lines` caps how
+many lines are read per window before deduplication, 500 by default. Check
+your own figure with `--print` before enabling the timer.
+
+#### Privacy
+
+What leaves the node is masked, deduplicated log templates plus per-module
+counts. A template still carries the fixed text of the log messages it stands
+for — that text is the signal — but the variable parts are replaced and
+identical events collapse into one counted entry, so no line is sent verbatim.
+Two passes run before anything is sent:
+`imageroot/pypkg/insights/scrub.py` removes likely secrets
+(`password=`, `token=`, `api_key=`, `Authorization` headers, long base64
+runs, email addresses), and `imageroot/pypkg/insights/masking.py` replaces
+variable text (timestamps, PIDs, addresses, UUIDs and similar) so that
+repeated events collapse to one template. This is defence in depth, not a
+guarantee.
+
+The destination is the Nethesis insights service, authenticated with the
+subscription identity (`system_id` and secret) the node already holds —
+nothing new to provision or store. This is an explicit improvement over the
+previous design: no third-party LLM API key is stored on any node any more,
+and `state/secrets.env` no longer exists.
+
+The feature is disabled by default.
+
 ## Uninstall
 
 To uninstall the instance:

docs/dev-environment-restore.md

@@ -0,0 +1,104 @@
+# Restoring the rl1 dev environment
+
+Steps to rebuild the `rl1.leader.default.gs.nethserver.net` dev box back to
+the state it was in during the `anomaly_detector` branch work, after it gets
+torn down. Written for an agent with no memory of this session.
+
+**Deliberately excluded** (ask the user/operator, never store in this repo):
+the cluster subscription `auth_token`, and any `system_id` value read back
+from `cluster/subscription`. Everything else below is reproducible from
+public repo state.
+
+## 1. Provision + install NS8
+
+Use the `accessing-nethserver-test-vps` skill (or, if unavailable,
+`ns8-terraform-infra`'s `tofu apply -var 'leader_node={"dn1":"rl1"}'`, then
+NS8 core install + `create-cluster`). Set the real admin password per that
+skill — do not leave the default cluster-admin password from
+`create-cluster` in place.
+
+## 2. Enroll the cluster subscription
+
+Required for the insights feature below to actually authenticate — without
+it, `insights-collector` logs "no subscription found" and ships nothing.
+
+```bash
+api-cli run cluster/set-subscription --data '{"subscription":{"auth_token":"<TOKEN>"}}'
+```
+
+`<TOKEN>` is a Nethesis subscription auth token, ≥32 chars — get it from the
+operator, not from any file in this repo. Confirm it landed with:
+
+```bash
+api-cli run cluster/get-subscription
+```
+
+## 3. Install modules
+
+```bash
+add-module ghcr.io/nethserver/loki:latest 1        # -> loki1
+add-module ghcr.io/nethserver/crowdsec:latest 1    # -> crowdsec1 (+ its firewall-bouncer companion)
+```
+
+`crowdsec1` was present on the box but **untouched** in this session — no
+config changes were made to it. It matters only if continuing the
+blocked-IP-evidence work; see
+`/home/giacomo/projects/ns8/ns8-crowdsec/crowdsec.plan` on this machine (not
+in this repo, not on rl1 — a local planning note) for that follow-on design.
+
+## 4. Update loki1 to the branch build
+
+The `anomaly_detector` branch publishes its image via CI on every push —
+no local build needed. Command actually used:
+
+```bash
+update-module ghcr.io/nethserver/loki:anomaly_detector loki1 --force
+```
+
+## 5. Configure the insights collector
+
+```bash
+api-cli run module/loki1/set-insights --data '{
+  "active": true,
+  "base_url": "https://controller.gs.nethserver.net/insights",
+  "verify_tls": false
+}'
+```
+
+The `/insights` path suffix is required — the server on
+`controller.gs.nethserver.net` is path-mounted, not on the bare host (bare
+host `/v1/bundles` 404s; `curl -k https://controller.gs.nethserver.net/insights/healthz`
+should return `200`). `verify_tls: false` matches that server's self-signed
+cert; use `true` against a properly-certified endpoint.
+
+## 6. Verify
+
+```bash
+# Confirm config landed
+api-cli run module/loki1/get-configuration
+# -> "insights": {"status": "active", "base_url": "https://controller.gs.nethserver.net/insights", ...}
+
+# Zero-cost payload check, no shipping
+runagent -m loki1 ../bin/insights-collector --print
+
+# Actually ship one window
+runagent -m loki1 ../bin/insights-collector
+# or, as the timer would:
+runagent -m loki1 systemctl --user start insights-collector.service
+runagent -m loki1 journalctl --user -u insights-collector
+```
+
+## Known state at time of writing (not yet resolved — do not assume fixed)
+
+- **Real end-to-end shipping against `controller.gs.nethserver.net` was not
+  confirmed working.** The server accepts auth and rejects malformed bodies
+  fast, but a real, well-formed bundle causes a request that hangs until the
+  client's 60s read timeout — looks like a server-side issue (likely a
+  downstream queue/broker not responding in that dev deployment), not
+  something fixable from `ns8-loki`. Re-check this before relying on it.
+- Local unit tests: `./test-unit.sh` → 106 passed.
+- CI on the branch: green as of commit `8b3e86b` (port-collision fix in
+  `tests/20__insights.robot` — the e2e stub was colliding with
+  node_exporter's default port 9100 on the test node; moved to 19100).
+- PR: https://github.com/NethServer/ns8-loki/pull/70 (draft, no assignee, no
+  reviewer requested as of last check).