Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,65 @@ You can also modify settings with the configure-module action
- `enable_online_api`: enable/disable to push signals and receive bad IPs from crowdsec hub (true/false default is true)
- `ban_local_network`: enable/disable to ban on private IP address range

## Push blocked-IP evidence to nethesis-insights

Every ban decision can be pushed in near-real-time to
[`nethesis-insights`](https://github.com/nethesis/nethesis-insights), so a
central service can aggregate them into a fleet-wide blacklist. Delivery uses
CrowdSec's own `notification-http` plugin, driven by `profiles.yaml`: no
polling, no cursor, no extra timer. Decisions fired within the same 30s window
are batched into a single `POST /v1/blocklist-evidence`.

api-cli run module/crowdsec1/set-insights --data '{
"active": true,
"base_url": "https://insights.example.com",
"verify_tls": true
}'

| Parameter | Env var | Required | Default |
|---|---|---|---|
| `active` | — | yes | `false` |
| `base_url` | `INSIGHTS_SERVER_URL` | when `active` | unset |
| `verify_tls` | `INSIGHTS_VERIFY_TLS` | no | `true` |

Disable it again with:

api-cli run module/crowdsec1/set-insights --data '{"active": false}'

No API key is required or accepted. Identity is not configurable: the
`notifications/http.yaml` render reads `system_id` and `auth_token` from the
`cluster/subscription` Redis hash and sends
`Authorization: Basic base64(system_id:auth_token)`. The credential is never
stored in the module environment, so a configured webhook cannot be pointed at
another tenant by editing module state, and a subscription registered later
starts working on the next reload with no reconfiguration. When the
subscription is terminated the webhook configuration is cleared.

`verify_tls: false` exists for self-signed test servers only.

Delivery is best effort: a decision made while `crowdsec1` is mid-restart, or
lost to a plugin subprocess crash, is not retried.

## get-configuration

Display the configuration

api-cli run get-configuration --agent module/crowdsec1 | jq

The `insights` block reports the webhook state:

```json
"insights": {
"status": "active",
"base_url": "https://insights.example.com",
"verify_tls": true,
"subscription_configured": true
}
```

`subscription_configured` is what tells the UI why an active webhook is
shipping nothing.

## Disable whitelist

By default whitelist is enabled to never ban IP on the local network, for test purpose you could disable it
Expand Down
12 changes: 12 additions & 0 deletions imageroot/actions/get-configuration/20read
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,16 @@ config["group_threshold"] = int(os.getenv("GROUP_THRESHOLD", 100))
config['dynamic_bantime_duration'] = os.getenv("DYNAMIC_BANTIME_DURATION", "4")
config['pull_community_blocklist'] = os.getenv("PULL_COMMUNITY_BLOCKLIST", "True") == "True"

# There is no timer here: the webhook is active as soon as the URL is set and
# the crowdsec notification plugin is wired up by expand-configuration.
# subscription_configured is what tells the UI why an active webhook is
# shipping nothing: identity comes from cluster/subscription, not from the
# module configuration.
config["insights"] = {
"status": "active" if os.getenv("INSIGHTS_SERVER_URL") else "inactive",
"base_url": os.getenv("INSIGHTS_SERVER_URL", ""),
"verify_tls": os.getenv("INSIGHTS_VERIFY_TLS", "1") != "0",
"subscription_configured": bool(agent.redis_connect().hgetall("cluster/subscription")),
}

json.dump(config, fp=sys.stdout)
44 changes: 42 additions & 2 deletions imageroot/actions/get-configuration/validate-output.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,13 @@
"enroll_instance": "cl7ze8xdn00030vl70tyutuxjj",
"group_threshold": 100,
"dynamic_bantime_duration": "4",
"pull_community_blocklist": true
"pull_community_blocklist": true,
"insights": {
"status": "active",
"base_url": "https://insights.example.com",
"verify_tls": true,
"subscription_configured": true
}
}
],
"type": "object",
Expand All @@ -37,7 +43,8 @@
"enable_online_api",
"ban_local_network",
"enroll_instance",
"group_threshold"
"group_threshold",
"insights"
],
"properties": {
"group_threshold": {
Expand Down Expand Up @@ -125,6 +132,39 @@
"type": "boolean",
"title": "pull_community_blocklist",
"description": "Pull the CrowdSec community blocklist from the Central API"
},
"insights": {
"type": "object",
"title": "insights",
"description": "State of the webhook that pushes ban decisions to the nethesis-insights service.",
"required": [
"status",
"base_url",
"verify_tls",
"subscription_configured"
],
"properties": {
"status": {
"type": "string",
"enum": [
"active",
"inactive"
],
"description": "Whether the blocklist-evidence webhook is configured."
},
"base_url": {
"type": "string",
"description": "Base URL of the nethesis-insights server that receives the decisions."
},
"verify_tls": {
"type": "boolean",
"description": "Whether the server TLS certificate is verified."
},
"subscription_configured": {
"type": "boolean",
"description": "True when cluster/subscription holds identity data. An active webhook with no subscription ships nothing."
}
}
}
}
}
37 changes: 37 additions & 0 deletions imageroot/actions/set-insights/10set
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/usr/bin/env python3

#
# Copyright (C) 2026 Nethesis S.r.l.
# SPDX-License-Identifier: GPL-3.0-or-later
#

import json
import os
import subprocess
import sys

import agent

request = json.load(sys.stdin)

if request['active']:
# A missing subscription does not abort the action: expand-configuration
# re-reads cluster/subscription at every render, so a subscription
# registered later starts working with no reconfiguration here.
subscription = agent.redis_connect().hgetall('cluster/subscription')
if not subscription:
print(agent.SD_WARNING + "No subscription found: decisions will not ship until the node has a subscription",
file=sys.stderr)

agent.set_env('INSIGHTS_SERVER_URL', request['base_url'].rstrip('/'))
agent.set_env('INSIGHTS_VERIFY_TLS', '1' if request.get('verify_tls', True) else '0')
else:
agent.munset_env(['INSIGHTS_SERVER_URL', 'INSIGHTS_VERIFY_TLS'])

# Reload, not restart: the unit's ExecReload already re-renders the
# configuration and sends SIGHUP to crowdsec, without dropping the container.
subprocess.run(["systemctl", "reload", f"{os.environ['MODULE_ID']}.service"],
stdout=sys.stderr,
stderr=sys.stderr,
text=True,
check=True)
40 changes: 40 additions & 0 deletions imageroot/actions/set-insights/validate-input.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"$schema": "http://json-schema.org/draft-04/schema#",
"$id": "http://schema.nethserver.org/crowdsec/set-insights.json",
"title": "Configure the insights blocklist-evidence webhook",
"description": "Configure the CrowdSec notification plugin that pushes every ban decision to the nethesis-insights service.",
"type": "object",
"properties": {
"active": {
"type": "boolean",
"description": "Enable or disable the insights webhook."
},
"base_url": {
"type": "string",
"format": "uri",
"description": "Base URL of the nethesis-insights server that receives the decisions."
},
"verify_tls": {
"type": "boolean",
"default": true,
"description": "Verify the server TLS certificate. Disable only when pointing at a self-signed test server."
}
},
"oneOf": [
{
"properties": {
"active": {"enum": [true]}
},
"required": [
"active",
"base_url"
]
},
{
"properties": {
"active": {"enum": [false]}
},
"required": ["active"]
}
]
}
45 changes: 45 additions & 0 deletions imageroot/bin/expand-configuration
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
# SPDX-License-Identifier: GPL-3.0-or-later
#

import base64
import json
import os
import agent
import re
Expand Down Expand Up @@ -88,6 +90,48 @@ if True:
with open("crowdsec_config/local_api_credentials.yaml.local","w") as f:
f.write(output)

# expand notifications/http.yaml, the nethesis-insights blocklist-evidence
# webhook. The identity is never stored in the module environment: it is
# re-read from cluster/subscription at every render, so a subscription
# registered later starts working with no reconfiguration.
insights_url = os.environ.get("INSIGHTS_SERVER_URL", "").rstrip("/")

files = ["crowdsec_config/notifications/http.yaml"]
for f in files:
try:
os.remove(f)
except FileNotFoundError:
pass

http_notification = False
if insights_url:
subscription = agent.redis_connect().hgetall('cluster/subscription')
system_id = subscription.get('system_id')
auth_token = subscription.get('auth_token')
if not system_id or not auth_token:
# Log the field names only: a value here would be the credential
# itself, and this output lands in the journal.
print(agent.SD_WARNING + "cluster/subscription is missing required fields, "
f"the insights webhook is not configured; present: {sorted(subscription.keys())}",
file=sys.stderr)
else:
properties = {
"insights_url": insights_url,
# Built in Python: Jinja has no base64 filter, and a filter chain
# is one templating bug away from leaking the token into an error
# message.
"basic_token": base64.b64encode(f"{system_id}:{auth_token}".encode()).decode(),
"system_id_json": json.dumps(system_id),
"insights_verify_tls": os.environ.get("INSIGHTS_VERIFY_TLS", "1") != "0",
}
os.makedirs("crowdsec_config/notifications", exist_ok=True)
template = jenv.get_template('http.yaml')
output = template.render(properties)
with open("crowdsec_config/notifications/http.yaml", "w") as f:
f.write(output)
os.chmod("crowdsec_config/notifications/http.yaml", 0o600)
http_notification = True

# expand profiles.yaml.local
smtp = agent.get_smarthost_settings(agent.redis_connect())
receiver_emails = os.environ.get("RECEIVER_EMAILS", "")
Expand All @@ -101,6 +145,7 @@ for f in files:

properties = {
"email": True if smtp['enabled'] and receiver_emails else False,
"http": http_notification,
"bantime": os.environ.get('BANTIME','1')+'m',
"dyn_bantime": os.environ.get('DYN_BANTIME',"True") == "True",
"dynamic_bantime_duration": os.environ.get('DYNAMIC_BANTIME_DURATION','4'),
Expand Down
20 changes: 20 additions & 0 deletions imageroot/events/subscription-changed/10insights
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/usr/bin/env python3

#
# Copyright (C) 2026 Nethesis S.r.l.
# SPDX-License-Identifier: GPL-3.0-or-later
#

import json
import sys

import agent

data = json.load(sys.stdin)

# A terminated subscription would otherwise leave the notification plugin
# pushing decisions authenticated with an identity that no longer validates.
# 20restart reloads the service right after this step, which re-renders
# notifications/http.yaml and drops it.
if data.get('action') == 'terminated':
agent.munset_env(['INSIGHTS_SERVER_URL', 'INSIGHTS_VERIFY_TLS'])
41 changes: 41 additions & 0 deletions imageroot/templates/http.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
type: http # Don't change
name: http_default # Must match the registered plugin in the profile

# One of "trace", "debug", "info", "warn", "error", "off"
log_level: info

# Batch the decisions fired within the same window into a single POST, so a
# burst (a distributed scan tripping the same scenario repeatedly) does not
# turn into one HTTP call per decision.
group_wait: 30s
timeout: 20s

#-------------------------
# plugin-specific options

# The following template receives a list of models.Alert objects and builds the
# nethesis-insights blocklist-evidence body. Fields are emitted through toJson
# so that the *string members of models.Decision are dereferenced and escaped
# by encoding/json instead of being printed as pointers.
{% raw %}
format: |
{{- $sep := "" -}}
{"schema_version":1,"system_id":{% endraw %}{{ system_id_json }}{% raw %},"decisions":[
{{- range . -}}
{{- $created := .CreatedAt -}}
{{- range .Decisions -}}
{{ $sep }}{"id":{{ .ID }},"value":{{ .Value | toJson }},"scope":{{ .Scope | toJson }},"type":{{ .Type | toJson }},"scenario":{{ .Scenario | toJson }},"origin":{{ .Origin | toJson }},"duration":{{ .Duration | toJson }},"created_at":{{ $created | toJson }}}
{{- $sep = "," -}}
{{- end -}}
{{- end -}}
]}
{% endraw %}

url: {{ insights_url }}/v1/blocklist-evidence
method: POST

headers:
Content-Type: application/json
Authorization: "Basic {{ basic_token }}"

skip_tls_verification: {{ "false" if insights_verify_tls else "true" }}
7 changes: 6 additions & 1 deletion imageroot/templates/profiles.yaml.local
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,13 @@ decisions:
- type: ban
duration: {{bantime}}
{% endif %}
{% if email %}
{% if email or http %}
notifications:
{% if email %}
- email_default # Set the required email parameters in /etc/crowdsec/notifications/email.yaml before enabling this.
{% endif %}
{% if http %}
- http_default # Pushes every decision to nethesis-insights, see /etc/crowdsec/notifications/http.yaml
{% endif %}
{% endif %}
on_success: break
Loading