Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Docs/implementation_plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ This pipeline spans two repositories:

| Repository | Purpose | Contents |
|---|---|---|
| **[ABEvalFlow](https://github.com/RHEcosystemAppEng/ABEvalFlow)** (this repo) | Pipeline definitions, scripts, templates, config | Tekton YAML, Python scripts, Jinja2 templates, Harbor backend |
| **[ABEvalFlow](https://github.com/RHEcosystemAppEng/agentic_eval_flow)** (this repo) | Pipeline definitions, scripts, templates, config | Tekton YAML, Python scripts, Jinja2 templates, Harbor backend |
| **[agentic-collections](https://github.com/RHEcosystemAppEng/agentic-collections)** | Skills, tasks, tests (post-evaluation) | Persona-based plugins (`rh-sre`, `rh-developer`, `ocp-admin`, etc.), 100+ skills |

The `tasks-treatment/` and `tasks-control/` directories generated during scaffolding are **ephemeral workspace artifacts** — they exist only during a pipeline run, not as permanent directories in either repo.
Expand Down
4 changes: 2 additions & 2 deletions Docs/konflux-integration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ WORKLOAD_NAMESPACE: "" # Required when EVAL_MODE=remote
WORKLOAD_CREDENTIALS_SECRET: "workload-cluster-credentials" # Secret name

# Pipeline repo (for evaluation scripts)
PIPELINE_REPO_URL: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
PIPELINE_REPO_URL: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
PIPELINE_REPO_REVISION: "main"
```

Expand Down Expand Up @@ -221,7 +221,7 @@ spec:
resourceKind: pipelinerun
params:
- name: url
value: https://github.com/RHEcosystemAppEng/ABEvalFlow
value: https://github.com/RHEcosystemAppEng/agentic_eval_flow
- name: revision
value: main
- name: pathInRepo
Expand Down
14 changes: 7 additions & 7 deletions Docs/manual_trigger_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@ A2A monitoring runs are triggered automatically by three sources (plus manual Pi
| Source | When it fires | Agent mode | Key params |
|--------|---------------|------------|------------|
| **Quay push webhook** | New image pushed to `quay.io/ecosystem-appeng/google-lightspeed-agent` | Ephemeral deploy | `agent-image`, `agent-tag`; leave `agent-endpoint` empty |
| **LiteLLM config change** | Push to `main` on `RHEcosystemAppEng/ABEvalFlow` modifying `config/litellm/*` | Existing deployed agent | `agent-endpoint` only |
| **LiteLLM config change** | Push to `main` on `RHEcosystemAppEng/agentic_eval_flow` modifying `config/litellm/*` | Existing deployed agent | `agent-endpoint` only |
| **10-day CronJob** | Scheduled every 10 days (`0 6 */10 * *`) | Existing deployed agent | `agent-endpoint` from canary pack config |

**Quay push:** EventListener trigger `quay-push-trigger` deploys a temporary instance of the pushed image, evaluates it, then cleans up. Requires a Quay repository notification pointing at the EventListener URL (pending Ilona access).

**LiteLLM config change:** EventListener trigger `litellm-config-push-trigger` re-tests against the existing agent at `http://lightspeed-agent.ab-eval-flow.svc:8000`. Requires a GitHub webhook on `RHEcosystemAppEng/ABEvalFlow` pointing at the EventListener URL.
**LiteLLM config change:** EventListener trigger `litellm-config-push-trigger` re-tests against the existing agent at `http://lightspeed-agent.ab-eval-flow.svc:8000`. Requires a GitHub webhook on `RHEcosystemAppEng/agentic_eval_flow` pointing at the EventListener URL.

**10-day CronJob:** The `abevalflow-monitoring` CronJob performs a health check on `/.well-known/agent.json` before triggering; skips eval if the agent is unhealthy.

Expand Down Expand Up @@ -156,7 +156,7 @@ spec:
name: abevalflow-monitoring-pipeline
params:
- name: repo-url
value: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
value: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: revision
value: "main"
- name: submission-dir
Expand Down Expand Up @@ -207,7 +207,7 @@ Fires when a new image is pushed to `quay.io/ecosystem-appeng/google-lightspeed-

### 2. LiteLLM Config Change

Fires when any file under `config/litellm/` (e.g., `config/litellm/configmap.yaml`) is pushed to the `main` branch of `RHEcosystemAppEng/ABEvalFlow`.
Fires when any file under `config/litellm/` (e.g., `config/litellm/configmap.yaml`) is pushed to the `main` branch of `RHEcosystemAppEng/agentic_eval_flow`.

- EventListener trigger: `litellm-config-push-trigger`
- Uses the existing deployed agent at `http://lightspeed-agent.ab-eval-flow.svc:8000`
Expand Down Expand Up @@ -237,7 +237,7 @@ spec:
name: abevalflow-monitoring-pipeline
params:
- name: repo-url
value: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
value: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: revision
value: "main"
- name: submission-dir
Expand Down Expand Up @@ -278,7 +278,7 @@ spec:
name: abevalflow-monitoring-pipeline
params:
- name: repo-url
value: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
value: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: revision
value: "main"
- name: submission-dir
Expand Down Expand Up @@ -586,7 +586,7 @@ spec:
name: abevalflow-pipeline
params:
- name: repo-url
value: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
value: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: revision
value: "main"
- name: submission-dir
Expand Down
2 changes: 1 addition & 1 deletion Docs/trigger_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -936,7 +936,7 @@ See `Docs/trigger_models_and_experiment_types.md` for details.

**Q: Who do I contact for help?**
Reach out to the ABEvalFlow team or open an issue in the
[ABEvalFlow repository](https://github.com/RHEcosystemAppEng/ABEvalFlow).
[ABEvalFlow repository](https://github.com/RHEcosystemAppEng/agentic_eval_flow).

---

Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# ABEvalFlow
# Agentic Eval Flow

Automated Tekton-orchestrated pipeline on OpenShift for evaluating AI artifacts:

Expand All @@ -10,7 +10,7 @@ Produces statistical reports with pass rates, uplift metrics, significance tests

## Pipelines

ABEvalFlow provides two pipeline variants:
Agentic Eval Flow provides two pipeline variants:

| Pipeline | Purpose | Key Differences |
|----------|---------|-----------------|
Expand Down Expand Up @@ -84,7 +84,7 @@ See [Gate Policy Configuration](#gate-policy-configuration) for full options.
## Repository Structure

```
ABEvalFlow/
agentic_eval_flow/
├── Docs/ # ADR, implementation plan, guides
├── pipeline/
│ ├── pipeline.yaml # Main pipeline definition
Expand All @@ -110,7 +110,7 @@ ABEvalFlow/
| Repository | Purpose |
|---|---|
| [skill-submissions](https://github.com/RHEcosystemAppEng/skill-submissions) | Submission intake — users push skills, MCP evals, and agent evals here |
| [skills_eval_corrections](https://github.com/RHEcosystemAppEng/skills_eval_corrections) | Harbor fork with OpenShift backend for ABEvalFlow |
| [skills_eval_corrections](https://github.com/RHEcosystemAppEng/skills_eval_corrections) | Harbor fork with OpenShift backend for Agentic Eval Flow |
| [All-Hands-AI/openhands-agent-monitor](https://github.com/All-Hands-AI/openhands-agent-monitor) | Harbor upstream — agent evaluation framework |
| [cisco-ai-defense/skill-scanner](https://github.com/cisco-ai-defense/skill-scanner) | Security scanner for prompt injection and data exfiltration detection |

Expand Down
2 changes: 1 addition & 1 deletion config/migrations/migration-job.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ spec:
set -e
REPO_REF="${PIPELINE_REPO_REVISION:-main}"
git clone --depth 1 --branch "$REPO_REF" \
https://github.com/RHEcosystemAppEng/ABEvalFlow.git /tmp/abevalflow
https://github.com/RHEcosystemAppEng/agentic_eval_flow.git /tmp/abevalflow
cd /tmp/abevalflow
pip install --quiet --no-cache-dir uv
uv sync --frozen
Expand Down
2 changes: 1 addition & 1 deletion config/mlflow/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# ABEvalFlow MLflow (eval-only)
# Agentic Eval Flow MLflow (eval-only)

Lightweight tracking server for AEH post-evaluate logging. **Not** production
telemetry HA — single replica, SQLite on a PVC.
Expand Down
4 changes: 2 additions & 2 deletions examples/a2a-agent-eval/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# A2A Agent Evaluation Example

This example demonstrates how to evaluate an A2A-compliant agent (like `google-lightspeed-agent`) using Harbor and the ABEvalFlow pipeline.
This example demonstrates how to evaluate an A2A-compliant agent (like `google-lightspeed-agent`) using Harbor and the Agentic Eval Flow pipeline.

## Prerequisites

Expand All @@ -26,7 +26,7 @@ harbor run \
--n-attempts 3
```

### Via ABEvalFlow Pipeline (Tekton)
### Via Agentic Eval Flow Pipeline (Tekton)

```bash
tkn pipeline start abevalflow-ci-pipeline \
Expand Down
4 changes: 2 additions & 2 deletions pipeline/integration/konflux-eval-pipelinerun.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ spec:
description: "Evaluation engine: harbor, ase, mcpchecker, a2a"
- name: SUBMISSION_REPO_URL
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: Git repo containing the submission definition
- name: SUBMISSION_DIR
type: string
Expand Down Expand Up @@ -127,7 +127,7 @@ spec:
# === Pipeline repo (for scripts) ===
- name: PIPELINE_REPO_URL
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: URL of the ABEvalFlow pipeline repo containing evaluation scripts
- name: PIPELINE_REPO_REVISION
type: string
Expand Down
2 changes: 1 addition & 1 deletion pipeline/pipelines/ci-pipeline-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ spec:
description: Submission directory name under submissions/
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/pipelines/ci-pipeline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ spec:
description: Submission directory name under submissions/
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/pipelines/monitoring-pipeline-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ spec:
description: Submission directory name under submissions/
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/pipelines/monitoring-pipeline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ spec:
description: Submission directory name under submissions/
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/runs/oracle_pipeline_run.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ spec:
- name: submission-dir
value: "sample-skill-ai"
- name: pipeline-repo-url
value: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
value: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
value: "APPENG-4909/ai-assisted-eval"
- name: enable-ai-generation
Expand Down
2 changes: 1 addition & 1 deletion pipeline/runs/oracle_test_run.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ spec:

# Clone the pipeline repo (which also has the example submission)
git clone --depth 1 --branch APPENG-4909/ai-assisted-eval \
https://github.com/RHEcosystemAppEng/ABEvalFlow.git _pipeline
https://github.com/RHEcosystemAppEng/agentic_eval_flow.git _pipeline

# Symlink example as a "submission" so the generate script finds it
mkdir -p submissions
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/components/ase-eval.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ spec:
description: Number of parallel eval cases within a single run
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: URL of the ABEvalFlow pipeline repository
- name: pipeline-repo-revision
type: string
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/components/generate_tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ spec:
description: Evaluation engine (harbor, ase, both)
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: URL of the ABEvalFlow pipeline repository
- name: pipeline-repo-revision
type: string
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/components/harbor-eval.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ spec:
description: Branch or SHA of the Harbor fork to install
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: URL of the ABEvalFlow pipeline repository
- name: pipeline-repo-revision
type: string
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/components/mcpchecker-eval.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ spec:
description: Pipeline run ID for provenance
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: URL of the ABEvalFlow pipeline repository
- name: pipeline-repo-revision
type: string
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/components/scaffold.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ spec:
description: Validated submission name (from validate-submission results)
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: URL of the ABEvalFlow pipeline repository
- name: pipeline-repo-revision
type: string
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/components/test-quality-review.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ spec:
description: Submission directory name under submissions/
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: URL of the ABEvalFlow pipeline repository
- name: pipeline-repo-revision
type: string
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/components/validate.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ spec:
description: Maximum allowed workspace size in MB (guards against oversized submissions)
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: URL of the ABEvalFlow pipeline repository
- name: pipeline-repo-revision
type: string
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/konflux/analyze-scorecard.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ spec:
recommendation. Set to 0.0 to pass whenever treatment >= control.
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: URL of the ABEvalFlow pipeline repository
- name: pipeline-repo-revision
type: string
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/konflux/evaluate.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ spec:
type: string
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/konflux/prepare.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ spec:
description: Evaluation engine (harbor, ase, mcpchecker, a2a, both)
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/konflux/red-team.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ spec:
default: ""
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/konflux/store.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ spec:
description: MinIO bucket for report storage
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
description: URL of the pipeline repo containing scripts
- name: pipeline-repo-revision
type: string
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/konflux/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ spec:
description: PipelineRun name for DB records
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ spec:
Degradation threshold as a ratio. Alert if current/previous < threshold.
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/phases/evaluate.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ spec:
description: PipelineRun name
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/phases/prepare.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ spec:
description: Treatment variant eval.yaml filename for AEH pairwise mode
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/phases/red-team.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ spec:
default: ""
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
2 changes: 1 addition & 1 deletion pipeline/tasks/phases/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ spec:
description: PipelineRun name for DB records
- name: pipeline-repo-url
type: string
default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git"
default: "https://github.com/RHEcosystemAppEng/agentic_eval_flow.git"
- name: pipeline-repo-revision
type: string
default: "main"
Expand Down
Loading
Loading