feat(insights): ship deduplicated log bundles to nethesis-insights

[gsanchietti]

Summary

The node stops analysing its own logs. ns8-loki now ships deduplicated log
bundles to a central analysis server, nethesis-insights,
and does nothing else with them.

This PR originally added an hourly detector that called an OpenAI-compatible
LLM from every node, with a configured API key, and wrote findings back to
the journal. That design does not survive a fleet: it puts a third-party
credential on 2700 machines, gives no cross-node memory, leaves cost
uncontrolled, and forces every prompt or schema change to land in two
implementations — a Python one at the edge and a Go one on the server — which
is how the two drift into different definitions of a finding.

The branch has been rewritten accordingly. The edge is replaced outright, not
dual-run: the LLM call, prompt rendering, findings parsing, webhook and
recall_findings() are gone from the node, and the API key with them.

Implements the cutover described in
docs/superpowers/specs/2026-08-05-nethesis-insights-design.md §14.

Related issue

None. This is a draft opened ahead of a formal issue.

What the node does now

A systemd timer fires a Type=oneshot service every 15 minutes. One run:

1. Takes the just-closed 15-minute window (compute_window() floors to the
   previous boundary, so there is no cursor file and no drift).
2. Queries local Loki over HTTP for a per-(module_id, priority) rate digest,
   a 7-day baseline, and up to --max-lines prefiltered lines
   (PRIORITY < 5 or category="security").
3. Scrubs likely secrets (scrub.py), masks volatile tokens (masking.py),
   and deduplicates the result into counted templates — measured at 7.1x
   reduction against six hours of real cluster logs.
4. POSTs the bundle to <base_url>/v1/bundles.

Gating, inference, finding identity and storage are all server-side.

Configuration

Three parameters, down from six:

api-cli run module/loki1/set-insights --data '{
  "active": true,
  "base_url": "https://insights.example.com",
  "verify_tls": false
}'

Parameter	Env var	Required	Default
active	— (enables insights-collector.timer)	yes	false
base_url	INSIGHTS_SERVER_URL	when active	unset
verify_tls	INSIGHTS_VERIFY_TLS	no	true

verify_tls: false exists for self-signed test servers. Parsing fails safe:
only 0, false, no and off disable verification; unset, empty and
unrecognised values all verify.

No API key is required or accepted any more. Identity is not configurable:
the collector reads system_id and auth_token from the cluster/subscription
Redis hash at each fire and sends Authorization: Basic base64(system_id:secret).
Two consequences worth stating — a configured collector cannot be pointed at
another tenant by editing module state, and a subscription registered after the
fact starts working with no reconfiguration. The server binds the bundle body's
system_id to the authenticated one and rejects a mismatch with 403.

Security

• No secret is stored by this module. state/secrets.env is deleted, along
  with its state-include.conf entry. The subscription credential is read from
  Redis per run and never written to disk, never passed through agent.set_env
  (which mirrors into module/<id>/environment), and never logged.
• When identity is missing, the collector logs the names of the fields
  present in cluster/subscription and never a value — enough for an operator
  to correct the node, not enough to leak a credential into the journal.
• Log text is attacker-influenced (public-internet sshd lines reach the
  bundle by design), so every line is flattened to one line and scrubbed before
  templating.
• The collector excludes its own SYSLOG_IDENTIFIER before the priority
  filter, so its diagnostics cannot feed back into the next window.
• --print needs neither a subscription nor a server URL and ships nothing, so
  an operator can see exactly what would leave the node before enabling
  anything.

Lifecycle

• subscription-changed (terminated) disables the timer and clears the
  environment — otherwise a terminated node keeps firing every 15 minutes
  against an identity that no longer validates.
• restore-module/95insights re-enables the timer when INSIGHTS_SERVER_URL
  came back, since timer enablement is not module state.
• update-module.d/15systemd tears down the never-released
  anomaly-detector.timer on dev nodes that installed an earlier branch build.

API changes

get-configuration returns an insights block in place of anomaly_detector:

"insights": {
  "status": "active",
  "base_url": "https://insights.example.com",
  "verify_tls": true,
  "subscription_configured": true,
  "last_run": "Wed 2026-08-07 14:00:11 UTC"
}

subscription_configured is what tells the UI why an enabled collector is
shipping nothing.

set-anomaly-detector is removed and replaced by set-insights. Nothing
consumed the old action outside this branch, which was never released.

How to test

# 1. Offline unit tests, in a container. No cluster needed.
./test-unit.sh

# 2. See exactly what would leave the node. No credentials, nothing shipped.
runagent -m loki1 ../bin/insights-collector --print

# 3. Point it at a server and fire one window by hand.
api-cli run module/loki1/set-insights --data '{
  "active": true, "base_url": "https://insights.example.com"
}'
runagent -m loki1 systemctl --user start insights-collector.service
runagent -m loki1 journalctl --user -u insights-collector

# 4. Confirm the timer and the reported state.
runagent -m loki1 systemctl --user list-timers insights-collector.timer
api-cli run module/loki1/get-configuration

# 5. Disable, and confirm the environment is cleared.
api-cli run module/loki1/set-insights --data '{"active": false}'

tests/20__insights.robot runs the same flow against a local stub server
(tests/insights-stub.py), asserting that the bundle carries a real
system_id, a Basic credential, and the injected synthetic line.

Verification

• 105 unit tests, offline, in a container, plus a CI job. Cover window
  flooring, the Basic credential, TLS context selection in both directions,
  the identity fallbacks, and the guarantee that a missing subscription logs
  field names but never values.
• CI green on this branch: unit tests, image publish, API schema docs.
• nethesis-insights server prototype verified end to end against this
  cluster's live Loki: template counts, security-category propagation, and
  idempotent re-delivery of the same window.

Known gaps before this leaves draft

• verify_tls is not yet proven in both directions against a real self-signed
  server. The flag is not tested until true actually rejects.
• imageroot/pypkg/insights/scrub.py has no dedicated unit test. Its
  Authorization-header, Bearer-token, email and long-blob rules are covered
  only incidentally by one assertion in test_bundle.py, so a regression in
  them would pass the suite silently. This is the code most directly
  responsible for keeping secrets out of a bundle and deserves its own tests.
• tests/20__insights.robot has not run in CI yet.