Update federated algorithms and controller endpoints

.gitignore

@@ -151,3 +151,10 @@ pyrightconfig.json
 /tests/test_data/.data_paths/
 /logs/
 poetry.toml
+
+# Project specific
+/runs/
+/.venv-play/
+/controller.local.toml
+/localworkers_config.json
+/status.txt

documentation/flower_phase1/DESIGN_DECISIONS.md

@@ -0,0 +1,39 @@
+# Design Decisions - Phase 1 (Simulation-First)
+
+## Scope
+The Phase 1 implementation provides federated training orchestration through Exaflow + Flower using simulation runtime.
+It does not include production hospital deployment, SuperLink/SuperNodes, or compliance hardening.
+
+## Core Architecture Split
+1. Exaflow = control plane.
+2. Flower = training/aggregation engine.
+3. Flower App = algorithm-specific training logic (flowertune_llm_medical).
+
+## Source of Truth Policy
+1. For Phase 1, source of truth for training data/runtime knobs is `parameters.dataset`.
+2. `inputdata` remains for compatibility and metadata.
+
+## Integration Strategy
+1. Do not vendor the full `adap/flower` repository.
+2. Integrate only the Flower app code as an Exaflow algorithm package.
+3. Keep Flower framework as a dependency.
+
+## Runtime Policy
+1. Only `runtime = simulation` is supported in Phase 1.
+2. The same API shape will extend to Phase 2 deployment runtime.
+3. `dataset` block semantics may change under `runtime=deployment` (local connectors instead of simulation partitioning).
+
+## Exaflow Boundary
+1. Exaflow does not manage model internals (LoRA weights, state dicts).
+2. Exaflow handles config, lifecycle status, metrics, and artifact references.
+
+## Contract Versioning
+1. `schema_version = "1.1"` for RunConfig and MetricsOut in this freeze.
+2. Backward-incompatible changes require a new major schema version.
+3. Backward-compatible additions may remain in the same major series.
+
+## Serialization Policy
+1. Boolean env values: `"true"` / `"false"`.
+2. List env values: JSON string.
+3. Optional `None`: env var omitted.
+4. `${request_id}` interpolation is resolved by Exaflow before dispatch.

documentation/flower_phase1/PHASE1_ACCEPTANCE.md

@@ -0,0 +1,37 @@
+# Phase 1 Acceptance Criteria
+
+## A. Smoke
+1. Job starts from Exaflow endpoint.
+2. Simulation completes at least 2 rounds without crash.
+3. Terminal status is produced.
+
+## B. Metrics
+1. At least one valid `round_event` per completed round.
+2. `requested_metrics` is always present.
+3. `reported_metrics` follows policy:
+   - RUNNING rounds: equals keys of `metrics`.
+   - failures: empty allowed.
+4. One `final_summary` is always emitted.
+5. Timestamps are serialized in UTC with `Z` suffix.
+
+## C. Artifacts
+1. `artifact_dir` resolves and is writable.
+2. Checkpoints are saved when configured.
+3. On `COMPLETED`, `artifacts.final_checkpoint_ref` exists.
+
+## D. Failure/Timeout/Cancel
+1. On `FAILED/CANCELLED/TIMEOUT`, `error` exists in final summary.
+2. `artifacts` is optional for non-completed final states.
+3. Partial/empty metrics payloads are accepted for failed terminal states.
+4. `CANCELLED` is best-effort stop and still emits terminal summary.
+
+## E. Contract Strictness
+1. Unknown RunConfig fields are rejected.
+2. Invalid combinations (e.g., `local_steps` + `epochs`) are rejected.
+3. Invalid `runtime != simulation` is rejected in Phase 1.
+4. Duplicate `request_id` follows idempotency policy (no duplicate run creation).
+
+## F. Non-Goals
+1. No real hospital connectors.
+2. No SuperLink/SuperNodes deployment runtime.
+3. No production security/compliance hardening in Phase 1.

documentation/flower_phase1/RUNTIME_CONTRACT.md

@@ -0,0 +1,65 @@
+# Runtime Contract - Phase 1
+
+## Trigger
+`POST /algorithms/flowertune_llm_medical`
+
+## Request Shape
+1. Exaflow standard `AlgorithmRequestDTO`.
+2. RunConfig is provided inside `parameters`.
+
+## RunConfig -> Env Mapping Rules
+1. Flatten nested config into env vars.
+2. booleans -> `"true"` / `"false"`.
+3. lists -> JSON strings.
+4. `None` -> omit env var.
+5. Canonical full naming for env vars (e.g., `MIN_EVALUATE_CLIENTS`).
+
+## Lifecycle
+1. `SUBMITTED`
+2. `RUNNING`
+3. Terminal: `COMPLETED` | `FAILED` | `CANCELLED` | `TIMEOUT`
+
+## Concurrency Scope (Phase 1)
+1. Single active Flower job is supported at a time.
+2. Flower helper endpoints are request-scoped via `request_id` query parameter.
+
+## Event Flow
+1. Exaflow receives and validates request.
+2. Exaflow creates job id and resolves artifact path.
+3. Exaflow dispatches Flower simulation run.
+4. Flower emits `round_event` payloads.
+5. Flower emits one terminal `final_summary` payload.
+6. Exaflow persists status, metrics, and artifacts references.
+
+## MetricsOut Status Semantics
+1. `round_event.status` is only `RUNNING` or `FAILED`.
+2. `COMPLETED` appears only in `final_summary.status`.
+3. `round_event.artifacts` object is always present; `checkpoint_ref` is optional.
+
+## Observability Fields
+1. `job_id` is required and the primary correlation key.
+2. `request_id` is optional but recommended.
+3. `correlation_id` is optional.
+4. `algorithm_name`, `runtime`, `timestamp` are required.
+5. Timestamps must be UTC with `Z` suffix.
+
+## Error Contract
+1. Error payload shape: `code`, `message`, optional `details`.
+2. `details` is intentionally open for runtime-specific diagnostics.
+3. Canonical Phase 1 error codes:
+   - `VALIDATION_ERROR`
+   - `ARTIFACT_PATH_ERROR`
+   - `MODEL_LOAD_ERROR`
+   - `DATASET_LOAD_ERROR`
+   - `QUANTIZATION_ERROR`
+   - `PEFT_CONFIG_ERROR`
+   - `ROUND_TIMEOUT`
+   - `JOB_TIMEOUT`
+   - `RUNTIME_ERROR`
+   - `CANCELLED_BY_USER`
+
+## Defaults Note
+`default` values in JSON Schema are informational only. Defaults are applied by producer/service logic, not by JSON Schema validators.
+
+## Cancellation Semantics
+Cancellation is best-effort stop for Phase 1 simulation. A terminal `final_summary` with `status=CANCELLED` is required.

documentation/flower_phase1/VALIDATION_SPEC.md

@@ -0,0 +1,52 @@
+# Validation Spec - Phase 1
+
+## Validation Layers
+1. Schema Validation (Pydantic).
+2. Service Validation (Exaflow business/runtime checks).
+3. Runtime Validation (Flower app startup/runtime checks).
+
+## A. Pydantic Validation (Request `parameters`)
+1. `extra = "forbid"` on all models.
+2. `runtime` must be `simulation`.
+3. `schema_version` must be `1.1`.
+4. `federation.num_rounds >= 1`, `num_clients >= 1`.
+5. `min_fit_clients <= num_clients`.
+6. `min_evaluate_clients <= num_clients`.
+7. `min_available_clients <= num_clients`.
+8. If `evaluation.enabled = false`: `fraction_evaluate = 0`, `min_evaluate_clients = 0`.
+9. `local_steps XOR epochs` (exactly one set).
+10. `fp16` and `bf16` cannot both be `true`.
+11. `seed >= 0`.
+12. `job_timeout_sec >= round_timeout_sec`.
+13. For `partitioner = iid`: `dataset.num_partitions = federation.num_clients`.
+14. If `peft.enabled = true`: `method`, `r`, `alpha` are required.
+15. Metrics allowlist for Phase 1: `loss`, `perplexity`.
+
+## B. Service Validation (Exaflow)
+1. Resolve `artifact_dir` template (`${request_id}`).
+2. Verify writable artifact path.
+3. Enforce capacity policy:
+   - max allowed `num_clients` for simulation,
+   - max allowed `model.max_seq_length`,
+   - allowed quantization modes,
+   - timeout upper bounds,
+   - optional GPU requirement policy.
+4. Normalize env map with canonical serialization rules.
+5. Enforce idempotency policy for duplicate `request_id` (return existing `job_id`).
+6. Enforce event invariants at ingestion:
+   - `round <= rounds_total`,
+   - `rounds_completed <= rounds_total`,
+   - `clients.participated <= clients.expected`,
+   - `clients.failed <= clients.expected`,
+   - `clients.participated + clients.failed <= clients.expected`,
+   - if present: `best_round <= rounds_completed`.
+7. Enforce metrics consistency policy:
+   - RUNNING round event: `reported_metrics == keys(metrics)`.
+   - FAILED/CANCELLED/TIMEOUT: empty metrics/report sets allowed.
+
+## C. Runtime Validation (Flower app)
+1. Model/tokenizer load success.
+2. Dataset load and partitioning success.
+3. Quantization compatibility.
+4. LoRA target modules validity.
+5. Early failure produces structured `ErrorInfo`.

exaflow/algorithms/federated/docs/glmm_binary.md

@@ -0,0 +1,74 @@
+## Binary GLMM (FederatedGLMMBinary)
+
+### Name
+
+**Binary GLMM (FederatedGLMMBinary)**
+
+### Type
+
+**Statistical Model** (binary outcome, random-intercept GLMM with Laplace approximation)
+
+### Goal (Why we need it)
+
+Estimate fixed effects and random-intercept variance for clustered binary outcomes in a federated setup, without exchanging row-level patient data.
+
+### When to use
+
+Use Binary GLMM when:
+
+* outcome is binary (`0/1`)
+* data are clustered by center/site
+* logistic regression without random effects is too optimistic
+* you need center-level heterogeneity modeling (`sigma_u2`)
+
+### When NOT to use
+
+Avoid / be careful when:
+
+* outcome is ordinal (use `FederatedGLMMOrdinal`)
+* no meaningful clustering exists
+* you need random slopes (not currently supported)
+* events are extremely rare with tiny per-center samples
+
+---
+
+### Inputs / Outputs
+
+| Item              | Description                                          |
+| ----------------- | ---------------------------------------------------- |
+| **X**             | Local feature matrix, shape `(n_local, p)`           |
+| **y**             | Local binary labels in `{0, 1}`                      |
+| **center_ids**    | Local cluster ids, shape `(n_local,)`                |
+| **fit_intercept** | Whether to add intercept term (default: `True`)      |
+| **max_iters**     | Maximum optimization iterations                       |
+| **tol_theta**     | Parameter-step tolerance                             |
+| **tol_score**     | Score-norm tolerance                                 |
+| **agg_client**    | Federated aggregation client                         |
+
+**Outputs (FederatedGLMMBinaryResults)**
+
+* `theta`: full vector `[beta..., log_sigma_u2]`
+* `params`: fixed effects `beta`
+* `sigma_u2`: random-intercept variance
+* `nobs`, `n_groups`
+* `converged`, `n_iter`
+* `predict(X)`: class probabilities
+* `history` (optional): optimization diagnostics
+
+### Key Differences from centralized GLMM
+
+| Aspect               | Centralized GLMM         | Exaflow FederatedGLMMBinary |
+| -------------------- | ------------------------ | --------------------------- |
+| Data access          | Full row-level data      | Local-only + aggregated stats |
+| Likelihood handling  | Direct centralized solve | Federated Newton updates with Laplace terms |
+| Random effects       | Broad options            | Random intercept             |
+| Compute topology     | Single node              | Multi-worker federation      |
+
+### Approximation vs Exactness
+
+| Component                    | Status in FederatedGLMMBinary     |
+| --------------------------- | --------------------------------- |
+| Fixed effects               | Iterative estimate                |
+| Random-intercept variance   | Iterative estimate (`sigma_u2`)   |
+| Likelihood treatment        | Laplace approximation             |
+| Random slopes               | Not supported                     |

exaflow/algorithms/federated/docs/glmm_ordinal.md

@@ -0,0 +1,78 @@
+## Ordinal GLMM (FederatedGLMMOrdinal)
+
+### Name
+
+**Ordinal GLMM (FederatedGLMMOrdinal)**
+
+### Type
+
+**Statistical Model** (ordered categorical outcome, random-intercept GLMM with Laplace approximation)
+
+### Goal (Why we need it)
+
+Model ordered outcomes (e.g., severity stages) with fixed effects, ordered cutpoints, and center-level random intercepts in a federated environment.
+
+### When to use
+
+Use Ordinal GLMM when:
+
+* outcome has natural order and `K >= 2` categories
+* data are clustered by center/site
+* proportional-odds style cumulative logit is appropriate
+* you need both fixed effects and random-intercept variability
+
+### When NOT to use
+
+Avoid / be careful when:
+
+* outcome is binary only (use `FederatedGLMMBinary`)
+* outcome is continuous (use `FederatedLMM`/`FederatedOLS`)
+* category order is not meaningful
+* you need random slopes (not currently supported)
+
+---
+
+### Inputs / Outputs
+
+| Item              | Description                                          |
+| ----------------- | ---------------------------------------------------- |
+| **X**             | Local feature matrix, shape `(n_local, p)`           |
+| **y**             | Local labels in `{0, ..., K-1}`                      |
+| **center_ids**    | Local cluster ids, shape `(n_local,)`                |
+| **K**             | Number of ordered categories                          |
+| **fit_intercept** | Whether to add intercept term (default: `True`)      |
+| **max_iters**     | Maximum optimization iterations                       |
+| **tol_theta**     | Parameter-step tolerance                             |
+| **tol_score**     | Score-norm tolerance                                 |
+| **agg_client**    | Federated aggregation client                         |
+
+**Outputs (FederatedGLMMOrdinalResults)**
+
+* `theta`: full parameter vector
+* `params`: fixed effects `beta`
+* `cutpoints`: ordered category thresholds
+* `sigma_u2`: random-intercept variance
+* `nobs`, `n_groups`
+* `converged`, `n_iter`
+* `predict(X)`: predicted class label
+* `predict_proba(X)`: class probabilities
+* `history` (optional): optimization diagnostics
+
+### Key Differences from centralized ordinal mixed models
+
+| Aspect               | Centralized model        | Exaflow FederatedGLMMOrdinal |
+| -------------------- | ------------------------ | ---------------------------- |
+| Data access          | Full row-level data      | Local-only + aggregated terms |
+| Ordinal structure    | Cumulative logit         | Cumulative logit             |
+| Random effects       | Broad options            | Random intercept             |
+| Compute topology     | Single node              | Multi-worker federation      |
+
+### Approximation vs Exactness
+
+| Component                    | Status in FederatedGLMMOrdinal   |
+| --------------------------- | -------------------------------- |
+| Fixed effects               | Iterative estimate               |
+| Cutpoints                   | Iterative estimate (ordered)     |
+| Random-intercept variance   | Iterative estimate (`sigma_u2`)  |
+| Likelihood treatment        | Laplace approximation            |
+| Random slopes               | Not supported                    |