feat: support charms that integrate with COS Lite

DESIGN.md

@@ -64,6 +64,10 @@ Supported:
 - hook tools needed by the charm
 - real Pebble in Docker
 - relations with virtual charms (see below)
+- multiple models (e.g. a charm model and a COS model)
+- cross-model relations via `juju offer` and `juju integrate <model>.<app>`
+- virtual bundles (e.g. `juju deploy cos-lite`)
+- `juju run` (actions) on virtual charms
 
 Not supported:
 
@@ -77,11 +81,19 @@ If a charm needs any of the above, use real Juju.
 
 `jjx` can recognize certain well-known charm names as "virtual" charms. A virtual charm has no charm code — `jjx` manages its workload and relation data directly.
 
+All virtual charms are registered in a central registry (`_virtual_registry.py`) that specifies their start function, relation populate function, endpoint metadata, display name, and teardown priority. Adding a new virtual charm requires only one registration call — no other file needs to change.
+
 Currently supported:
 
-- `postgresql-k8s` — starts a real PostgreSQL 16 container in Docker and provides the `postgresql_client` interface. When a charm integrates with it, `jjx` populates the relation databag and creates a Juju secret with the database credentials, mimicking what the real `postgresql-k8s` charm's `DatabaseProvides` would write. The charm under test sees real relation data and real secrets, and can connect to the running PostgreSQL instance.
+- `postgresql-k8s` — starts a real PostgreSQL 16 container and provides the `postgresql_client` interface.
+- `loki-k8s` — starts a real Loki container and provides the `loki_push_api` interface. Workload logs flow via real Pebble log-targets.
+- `prometheus-k8s` — starts a real Prometheus container and consumes the `prometheus_scrape` interface. Configures itself from the charm's relation data.
+- `grafana-k8s` — starts a real Grafana container and consumes the `grafana_dashboard` interface. Provisions Prometheus and Loki as datasources, imports dashboards from relation data.
+- `traefik-k8s` — a state-only virtual charm (no container) that responds to the `show-proxied-endpoints` action with the URLs of other COS charms.
+
+Virtual bundles (e.g. `cos-lite`) deploy multiple virtual charms in one `juju deploy` command.
 
-Virtual charm containers are named `<model>-<app>-postgres` and are cleaned up on model teardown alongside workload containers. They are removed first, before workload containers.
+Virtual charm containers are named `<model>-<app>` and are cleaned up on model teardown. Teardown order is determined by each charm's `teardown_priority`: COS containers (grafana=10, prometheus=20, loki=30) are removed first, then postgres (40), workload (50), and charm runners (60).
 
 ## runtime model
 
@@ -103,6 +115,8 @@ The `.charm` file passed to deploy is a trigger only. `jjx` does not inspect or
 - `./.jjx/charm/` (staged runtime charm directory with `src/`, `lib/`, `metadata.yaml`, `config.yaml`, and `.unit-state.db`)
 - `./.jjx/socket` (Pebble API Unix socket, bind-mounted into both the workload and charm runner containers)
 - `./.jjx/<app>.<pid>.deploy` (marker files for in-flight background pebble-ready processes; created by deploy, deleted by the process on completion or by teardown)
+- `./.jjx/prom-config-<app>/` (Prometheus config directory, bind-mounted into the Prometheus container)
+- `./.jjx/grafana-config-<app>/` (Grafana provisioning directory, bind-mounted into the Grafana container)
 
 `jjx` also caches the Pebble binary at `~/.cache/jjx/pebble-bin`, downloaded from canonical/pebble GitHub Releases on first use. This cache is shared across projects and persists across model teardowns to enable reuse across multiple deployments.
 
@@ -113,7 +127,7 @@ Notes on generated runtime files:
 - `JJX_STATE_DIR` is set to `/jjx` (the in-container mount point of `.jjx/`) in the charm runner environment. Hook tools call back into `jjx` to read and write state; this env var lets them locate state directly.
 - `./.jjx/charm/.unit-state.db` is created by charm runtime state persistence (written by `ops` via `sqlite3` to `JUJU_CHARM_DIR/.unit-state.db`, which inside the charm runner is `/charm/.unit-state.db`).
 
-When the model is torn down, jjx kills any background pebble-ready processes (via `.deploy` marker files), then removes the entire `./.jjx/` directory. The `~/.cache/jjx/pebble-bin` cache is kept for reuse across subsequent deployments.
+When a model is torn down, jjx kills any background pebble-ready processes for that model's apps (via `.deploy` marker files), removes its containers in priority order, and removes the model from state. When the last model is torn down, the entire `./.jjx/` directory is removed. The `~/.cache/jjx/pebble-bin` cache is kept for reuse across subsequent deployments.
 
 ### charm runner container
 
@@ -138,7 +152,7 @@ The charm runner is a persistent Docker container that executes charm hooks.
 
 **Entrypoint**: The container runs `python -c 'import time; time.sleep(999999)'` — it stays alive doing nothing. Charm hooks are executed via `docker exec`.
 
-**Teardown**: The charm runner is removed *last* during model teardown, after postgres and workload containers. This ensures it is available for any cleanup hooks that might need it.
+**Teardown**: The charm runner is removed *last* during model teardown (priority 60), after COS containers, postgres, and workload containers. This ensures it is available for any cleanup hooks that might need it.
 
 ## execution contract
 
@@ -160,6 +174,7 @@ Deploy flow:
 1. ensure `./.jjx` exists and load state
 2. stage runtime charm files in `./.jjx/charm/` (`src/`, `metadata.yaml`, `config.yaml`)
 3. start workload container and Pebble with explicit `--network bridge` (no host networking)
+   - Pebble is started with `run --hold --create-dirs` so baked-in layers (e.g. Rockcraft layers with `startup: enabled`) do not autostart services before the charm's pebble-ready hook fires — matching real Juju, which also uses `--hold`
    - if `JJX_PUBLISH` is set to `HOST_PORT:CONTAINER_PORT`, add port publish `127.0.0.1:HOST_PORT:CONTAINER_PORT`
 4. start charm runner container (`--network=container:<workload>`, bind-mounts for Python, venv, jjx source, charm dir, state dir)
 5. wait for Pebble socket (`/jjx/socket`) to be connectable inside charm runner
@@ -200,10 +215,11 @@ Config flow:
 Destroy flow:
 
 1. kill any background pebble-ready processes (via `.jjx/*.deploy` marker files)
-2. stop and remove postgres containers (if any)
-3. stop and remove workload containers
-4. stop and remove charm runner containers (last)
-5. remove `./.jjx/` directory
+2. remove containers in teardown-priority order: COS containers (grafana, prometheus, loki) → postgres → workload → charm runners
+3. clean up any stragglers (orphaned containers not tracked in state)
+4. remove this model from state; if it's the last model, remove `./.jjx/` entirely
+
+When `jjx down` tears down all models, models are destroyed in reverse creation order so that COS models (created later) are torn down first.
 
 ## behavior guarantees
 
@@ -214,6 +230,17 @@ Destroy flow:
 - deterministic single-unit semantics
 - charm code that connects to loopback addresses (`127.0.0.1`, `localhost`, `::1`) reaches the workload container without exposing container ports on the host (the charm runner shares the workload's network namespace)
 
+## additional juju commands
+
+jjx implements several juju commands that jubilant/pytest-jubilant may call during setup, teardown, or status checks. These are minimal stubs that return just enough data for jubilant to function:
+
+- `juju offer` — records a cross-model offer in model state
+- `juju run` — executes actions on virtual charms (e.g. traefik's `show-proxied-endpoints`)
+- `juju switch` — no-op (jjx always uses `--model`)
+- `juju version` — returns a minimal version response
+- `juju show-model` — returns model metadata
+- `juju models` — lists all models in state
+
 ## constraints
 
 - requires Docker

README.md

@@ -50,8 +50,8 @@ __init__.py
 app.py
 database.py
 <b>pebble:/#</b> services
-<b>SERVICE</b>          <b>STARTUP</b>  <b>CURRENT</b>
-fastapi-service  enabled  active
+<b>SERVICE</b>  <b>STARTUP</b>  <b>CURRENT</b>
+fastapi  enabled  active
 </pre>
 
 For more detail, see [Command reference](https://borescope.dev/docs/reference-commands.html) in the borescope docs.
@@ -77,7 +77,7 @@ This runs the charm's integration tests and starts a Docker container for the wo
 
 The workload stays running until you press Ctrl-C.
 
-The command output includes the IP address of the workload. You can play with the workload by connecting to this address.
+The output includes the IP address of the workload. You can play with the workload by connecting to this address.
 
 Alternatively, to play with the workload on localhost, specify a port mapping:
 

justfile

@@ -37,6 +37,10 @@ charms:
   pushd tests/functional/charms/k8s-4-action
   charmcraft fetch-libs
   popd
+  cp -r operator/examples/k8s-5-observe tests/functional/charms
+  pushd tests/functional/charms/k8s-5-observe
+  charmcraft fetch-libs
+  popd
   rm -rf operator
 
 [private]

src/jjx/__init__.py

@@ -2,6 +2,73 @@
 from ._cli import run_hook_tool
 from ._engine import _CONTAINER_BINARY
 
+# Register all virtual charms. This must happen before any command module
+# uses the registry. Importing here ensures registration on package import.
+from . import _virtual_registry as _vr
+from . import (
+    _virtual_postgres,
+    _virtual_loki,
+    _virtual_prometheus,
+    _virtual_grafana,
+    _virtual_traefik,
+)
+
+_vr.register(
+    _vr.VirtualCharmSpec(
+        kind="postgresql",
+        start=_virtual_postgres.start_postgres_wrapper,
+        populate=_virtual_postgres.populate_relation,
+        info_key="pg_info",
+        endpoints={"database": {"interface": "postgresql_client", "role": "provides"}},
+        teardown_priority=40,
+    )
+)
+_vr.register(
+    _vr.VirtualCharmSpec(
+        kind="loki",
+        start=_virtual_loki.start_loki,
+        populate=_virtual_loki.populate_relation,
+        info_key="loki_info",
+        endpoints={"logging": {"interface": "loki_push_api", "role": "provides"}},
+        display_name="Loki API",
+        default_port=3100,
+        teardown_priority=30,
+    )
+)
+_vr.register(
+    _vr.VirtualCharmSpec(
+        kind="prometheus",
+        start=_virtual_prometheus.start_prometheus,
+        populate=_virtual_prometheus.populate_relation,
+        info_key="prom_info",
+        endpoints={"metrics-endpoint": {"interface": "prometheus_scrape", "role": "requires"}},
+        display_name="Prometheus",
+        default_port=9090,
+        teardown_priority=20,
+    )
+)
+_vr.register(
+    _vr.VirtualCharmSpec(
+        kind="grafana",
+        start=_virtual_grafana.start_grafana,
+        populate=_virtual_grafana.populate_relation,
+        info_key="grafana_info",
+        endpoints={"grafana-dashboard": {"interface": "grafana_dashboard", "role": "requires"}},
+        display_name="Grafana",
+        default_port=3000,
+        teardown_priority=10,
+    )
+)
+_vr.register(
+    _vr.VirtualCharmSpec(
+        kind="traefik",
+        start=_virtual_traefik.start_traefik,
+        populate=lambda model_state, relation, provider_app, info: None,
+        info_key="traefik_info",
+        teardown_priority=50,
+    )
+)
+
 
 def container_runtime() -> str:
     return _CONTAINER_BINARY

src/jjx/_cli.py

@@ -21,8 +21,11 @@
     _cmd_deploy,
     _cmd_destroy_model,
     _cmd_integrate,
+    _cmd_misc,
+    _cmd_offer,
     _cmd_remove_application,
     _cmd_hook_tool,
+    _cmd_run,
     _cmd_status,
     _cmd_wait_for,
     _engine,
@@ -69,6 +72,18 @@ def run_juju_command(argv: list[str]) -> int:
         return _cmd_debug_log.debug_log(rest, model)
     if command == "destroy-model":
         return _cmd_destroy_model.destroy_model(rest)
+    if command == "offer":
+        return _cmd_offer.offer(rest, model)
+    if command == "run":
+        return _cmd_run.run(rest, model)
+    if command == "switch":
+        return _cmd_misc.switch(rest)
+    if command == "version":
+        return _cmd_misc.version(rest)
+    if command == "show-model":
+        return _cmd_misc.show_model(rest, model)
+    if command == "models":
+        return _cmd_misc.models(rest)
 
     raise _engine.CliError(f"unknown command: {command}")
 
@@ -99,10 +114,47 @@ def juju_cli() -> int:
 def teardown_all_models() -> None:
     """Destroy all models currently in state."""
     state = _engine._load_state()
-    for model_name in state.get("models", {}):
+    # Destroy models in reverse creation order so that COS models (created
+    # later by JujuFactory) are torn down before the charm's model. This
+    # produces a more natural teardown order: COS containers first.
+    for model_name in reversed(list(state.get("models", {}))):
         _cmd_destroy_model.destroy_model([model_name])
 
 
+def _cos_endpoint_lines() -> list[str]:
+    """Return human-readable endpoint lines for virtual COS charms across all models.
+
+    Uses the virtual charm registry to find charms with display names.
+    Returns lines like ``Loki         http://172.17.0.3:3100`` so the user can
+    interact with them directly in a browser or via curl.
+    """
+    from . import _virtual_registry
+
+    state = _engine._load_state()
+    endpoints: dict[str, str] = {}
+    for model_name, model_state in state.get("models", {}).items():
+        for app_name, app_state in model_state.get("apps", {}).items():
+            if not app_state.get("virtual"):
+                continue
+            virtual_kind = app_state.get("virtual_kind")
+            spec = _virtual_registry.get_spec(virtual_kind or "")
+            if spec is None or spec.display_name is None:
+                continue
+            info = app_state.get(spec.info_key, {})
+            url = _virtual_registry.resolve_endpoint_url(info, spec.default_port)
+            if url:
+                endpoints[spec.display_name] = url
+
+    # Return sorted by teardown_priority (grafana first, then prometheus, loki)
+    specs_with_endpoints = [
+        (s, endpoints[s.display_name])
+        for s in _virtual_registry._REGISTRY.values()
+        if s.display_name and s.display_name in endpoints
+    ]
+    specs_with_endpoints.sort(key=lambda x: x[0].teardown_priority)
+    return [f"{spec.display_name:<12} {url}" for spec, url in specs_with_endpoints]
+
+
 def jjx_pytest_env_args(charm_root: Path) -> list[str]:
     """Return uv-run args that keep jjx resolution consistent with launch mode.
 
@@ -231,7 +283,6 @@ def jjx_cli() -> int:
 
     # Extract -p flag for port publishing.
     publish = None
-    publish_output = ""
     if "-p" in jjx_args:
         idx = jjx_args.index("-p")
         if idx + 1 < len(jjx_args):
@@ -266,10 +317,7 @@ def jjx_cli() -> int:
     env.pop("VIRTUAL_ENV", None)
     if publish:
         env["JJX_PUBLISH"] = publish
-        external_port, internal_port = publish.split(":", 1)
-        publish_output = (
-            f"\n\nPublished container port {internal_port} to 127.0.0.1:{external_port}"
-        )
+        external_port, _ = publish.split(":", 1)
     cmd = [
         "uv",
         "run",
@@ -287,10 +335,18 @@ def jjx_cli() -> int:
         if container is None:
             teardown_all_models()
             return proc.returncode
-        print(
-            f"\nStarted workload container {container.name} with IP {container.ip_address}{publish_output}",
-            flush=True,
-        )
+        if publish:
+            workload_line = f"Workload running at 127.0.0.1:{external_port}"
+        else:
+            workload_line = f"Workload running at {container.ip_address}"
+        print(f"\n{workload_line}", flush=True)
+        # List user-facing endpoints for virtual COS charms (loki, etc.)
+        # so the user can interact with them directly.
+        cos_lines = _cos_endpoint_lines()
+        if cos_lines:
+            print(flush=True)
+            for line in cos_lines:
+                print(line, flush=True)
         if detach:
             print("\nRun 'jjx down' to tear down")
             return proc.returncode

src/jjx/_cmd_add_model.py

@@ -16,8 +16,6 @@ def add_model(args: list[str]) -> int:
     model_name = filtered[0]
     state = _engine._load_state()
     models = state.setdefault("models", {})
-    if models and model_name not in models:
-        raise _engine.CliError("only a single model is supported")
     if model_name in models:
         raise _engine.CliError(f"model {model_name} already exists")