Add Railway deployment and Supabase persistence

.github/copilot-instructions.md

@@ -0,0 +1,125 @@
+# RD-Agent Copilot Instructions
+
+RD-Agent is a Microsoft Research R&D automation framework that automates the **Research (propose hypotheses) → Development (implement)** cycle for data-driven ML tasks. Primary scenarios: quantitative finance (factor/model development via Qlib), data science/Kaggle competitions, and LLM fine-tuning.
+
+## Build, Test & Lint
+
+### Setup
+```bash
+make dev          # install all optional deps + pre-commit hook
+make install      # editable install only
+```
+
+### Testing
+```bash
+make test                        # run all tests with coverage
+make test-offline                # offline tests only (no LLM API calls)
+pytest test/path/to/test_file.py::TestClass::test_method   # single test
+pytest -m offline                # run only @pytest.mark.offline tests
+```
+
+Mark tests that don't call external APIs with `@pytest.mark.offline`. The `workspace/` directory is excluded from test discovery. Coverage threshold is 80%.
+
+### Linting
+```bash
+make lint         # mypy + ruff + isort + black + toml-sort (check only)
+make auto-lint    # auto-fix isort + black + toml-sort
+make mypy         # type check rdagent/core (scope is expanding)
+make ruff         # lint rdagent/core
+```
+
+- **Line length:** 120 (black + ruff)
+- **isort profile:** black
+- **mypy:** strict (`disallow_untyped_defs=true`, `warn_return_any=true`) — currently scoped to `rdagent/core/`
+- Conventional commit prefixes required on PRs: `feat:`, `fix:`, `refactor:`, `test:`, `docs:`, `chore:`, etc.
+
+## Architecture
+
+### Package Layout
+
+```
+rdagent/
+├── core/          # Abstract base classes and framework interfaces
+├── components/    # Reusable building blocks (coder, runner, proposal, workflow, …)
+├── scenarios/     # Domain-specific implementations (qlib, kaggle, data_science, …)
+├── app/           # Entry points per scenario (qlib_rd_loop, data_science, kaggle, …)
+├── oai/           # LLM backend abstraction (LiteLLM default, OpenAI, Azure)
+├── log/           # Logging (loguru-based), Streamlit trace viewer UI
+└── utils/         # Env execution, workflow loop, repo/blob utilities
+```
+
+### The R&D Loop
+
+Each scenario runs a `LoopBase` (via `LoopMeta`) that orchestrates these steps:
+
+```
+HypothesisGen → Hypothesis2Experiment → Developer (coder) → Developer (runner) → Experiment2Feedback
+     ↑                                                                                    |
+     └────────────────────── Trace (accumulates history) ───────────────────────────────┘
+```
+
+`RDLoop` in `rdagent/components/workflow/rd_loop.py` is the canonical implementation. Each component is dynamically loaded by class path from a `BasePropSetting` config object.
+
+### Core Abstractions (`rdagent/core/`)
+
+| Class | Role |
+|---|---|
+| `Scenario` | Domain context — provides background, runtime environment description |
+| `Developer[Exp]` | Transforms an experiment **in-place** (coder or runner) |
+| `Evaluator` / `IterEvaluator` | Produces `Feedback`; iter variant uses coroutine `yield`/`send` |
+| `EvolvingStrategy[T]` | Algorithm for one evolution iteration; yields partial states |
+| `RAGStrategy[T]` | Manages a `EvolvingKnowledgeBase`: query, generate, dump, load |
+| `Hypothesis` | Research idea with `hypothesis`, `reason`, `concise_*` fields |
+| `ExperimentFeedback` | Boolean `decision` + `reason`; `bool(fb)` is `fb.decision` |
+| `Workspace[Task, FB]` | Mutable container for code/data during task execution |
+| `EvoStep[T]` | Dataclass: `evolvable_subjects`, `queried_knowledge`, `feedback` |
+
+**Critical Developer contract:** `develop(exp)` mutates `exp` in-place. Do not return a new object.
+
+### Configuration
+
+All settings use **Pydantic v2 `BaseSettings`** via `ExtendedBaseSettings` (supports env-prefix inheritance across base classes):
+
+```python
+from rdagent.core.conf import ExtendedBaseSettings, RD_AGENT_SETTINGS
+from rdagent.oai.llm_conf import LLM_SETTINGS
+```
+
+Settings are overridable via environment variables or a `.env` file in the working directory (auto-loaded by the CLI). Key settings:
+- `RD_AGENT_SETTINGS.workspace_path` — where experiment artifacts are written (`git_ignore_folder/` by default)
+- `LLM_SETTINGS.backend` — LLM provider class path (default: `rdagent.oai.backend.LiteLLMAPIBackend`)
+- `LLM_SETTINGS.chat_model` / `embedding_model`
+
+### LLM Access
+
+```python
+from rdagent.oai.llm_utils import APIBackend
+response = APIBackend().build_messages_and_create_chat_completion(...)
+embeddings = APIBackend().create_embedding(str_list)
+```
+
+`APIBackend` is a factory that returns the configured backend. LLM calls are cached via MD5-keyed pickle when `LLM_SETTINGS.use_chat_cache = True`.
+
+### Prompts
+
+Prompts live in YAML files alongside the module that uses them. They are loaded via `Prompts(file_path)` (a `SingletonBaseClass` dict). Jinja2-style templating is used for variable substitution. Prompt files are typically at `rdagent/{component}/prompts.yaml`.
+
+### Logging
+
+```python
+from rdagent.log import rdagent_logger as logger
+logger.info("message")
+logger.log_object(obj, tag="label")   # structured object logging
+```
+
+`rdagent_logger` is a global `RDAgentLog` singleton backed by loguru. Use `logger.log_object()` for structured data (scenarios, settings, experiments). The Streamlit UI (`rdagent ui`) reads these logs for visualization.
+
+## Key Conventions
+
+- **`from __future__ import annotations`** at the top of every module — required for forward references with mypy.
+- **TypeVars are bound**: `ASpecificExp = TypeVar("ASpecificExp", bound=Experiment)`. Follow this pattern when adding generics.
+- **`SingletonBaseClass`**: kwargs-only construction enforced. Do not pass positional args to singletons.
+- **`import_class(dotted.path)`** from `rdagent.core.utils` — used everywhere to dynamically load scenario components from config strings. Use this rather than direct imports when the class path is user-configurable.
+- **`git_ignore_folder/`** — runtime workspace output; already gitignored. Experiment artifacts, pickle caches, and checkpoints go here.
+- **Parallel loops**: `RD_AGENT_SETTINGS.step_semaphore` controls concurrency. When `> 1`, subprocesses are used automatically.
+- **Ruff ignore list**: `ANN401`, `D` (docstrings), `ERA001`, `T20` (print), `S101` (assert), `TD`/`FIX` (todos) are intentionally suppressed project-wide.

Dockerfile

@@ -0,0 +1,33 @@
+FROM node:20-bookworm-slim AS frontend-builder
+
+WORKDIR /app/web
+COPY web/package.json web/package-lock.json ./
+RUN npm install --legacy-peer-deps
+COPY web/ ./
+RUN npm run build:flask
+
+
+FROM python:3.11-slim AS runtime
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1 \
+    SETUPTOOLS_SCM_PRETEND_VERSION_FOR_RDAGENT=0.0.0 \
+    PYTHONPATH=/app
+
+WORKDIR /app
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    git \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY . /app
+COPY --from=frontend-builder /app/git_ignore_folder/static /app/git_ignore_folder/static
+
+RUN python -m pip install --upgrade pip setuptools wheel && \
+    python -m pip install .
+
+EXPOSE 19899
+
+CMD sh -c 'python -m rdagent.app.cli server_ui --port "${PORT:-19899}"'

README.md

@@ -376,6 +376,10 @@ rdagent server_ui --port 19899
 
 After that, open `http://127.0.0.1:19899` in your browser.
 
+#### Railway Deployment
+
+The repository now includes `nixpacks.toml` and `railway.json` for deploying `rdagent server_ui` on Railway. If you want uploads, trace artifacts, and stdout logs to survive container replacement, enable the Supabase-backed persistence settings documented in [docs/deployment/railway.md](docs/deployment/railway.md).
+
 #### Common Notes
 
 Port `19899` is used in the examples above. Before starting either UI, check whether this port is already occupied. If it is, please change it to another available port.

docs/deployment/railway.md

@@ -0,0 +1,116 @@
+# Deploying RD-Agent to Railway with Supabase
+
+This repository now includes a Railway-ready `nixpacks.toml` and `railway.json` for deploying the Flask-backed Web UI:
+
+- backend entrypoint: `rdagent server_ui`
+- frontend build: `web/` via `npm run build:flask`
+- persistent artifacts: optional Supabase-backed upload/trace/stdout sync
+
+## What this deployment supports today
+
+This deployment path runs the existing `server_ui` service on Railway.
+
+- The Vue frontend is built into `git_ignore_folder/static`
+- Flask serves the built frontend and real-time APIs
+- RD-Agent jobs still run as subprocesses launched by the Flask service
+- If Supabase persistence is enabled, uploaded files, trace artifacts, and stdout logs are mirrored to Supabase Storage and can be hydrated back after container replacement
+
+This is the shortest path to getting the current project online on Railway without rewriting the execution model.
+
+## 1. Create the Railway service
+
+1. Create a new Railway project from this repository.
+2. Keep the service root at the repository root.
+3. Railway will pick up `nixpacks.toml` automatically.
+
+The service starts with:
+
+```bash
+rdagent server_ui --port $PORT
+```
+
+## 2. Create Supabase resources
+
+Create one Supabase project and at least one Storage bucket for RD-Agent artifacts.
+
+Recommended setup:
+
+- **Postgres**: optional for future task metadata / queue state
+- **Storage bucket**: required for persisted artifacts
+
+Suggested bucket name:
+
+```text
+rdagent
+```
+
+## 3. Configure Railway environment variables
+
+### Required for the web service
+
+```bash
+PORT                         # Provided by Railway
+UI_STATIC_PATH=./git_ignore_folder/static
+UI_TRACE_FOLDER=./git_ignore_folder/traces
+```
+
+### Enable Supabase-backed artifact persistence
+
+```bash
+UI_SUPABASE_ENABLED=true
+UI_SUPABASE_URL=https://<your-project-ref>.supabase.co
+UI_SUPABASE_SERVICE_ROLE_KEY=<supabase-service-role-key>
+UI_SUPABASE_BUCKET=rdagent
+```
+
+Optional path prefixes inside the bucket:
+
+```bash
+UI_SUPABASE_TRACE_PREFIX=traces
+UI_SUPABASE_STDOUT_PREFIX=stdout
+UI_SUPABASE_UPLOAD_PREFIX=uploads
+```
+
+### LLM / provider configuration
+
+Set the same provider credentials you already use locally. Common examples:
+
+```bash
+LLM_BACKEND=rdagent.oai.backend.LiteLLMAPIBackend
+OPENAI_API_KEY=<your-openai-key>
+OPENAI_API_BASE=<optional-base-url>
+CHAT_MODEL=<your-model-name>
+EMBEDDING_MODEL=<your-embedding-model>
+```
+
+If you use Azure or other providers, set the corresponding `LLM_SETTINGS` environment variables used by this repo.
+
+## 4. Deploy
+
+Once the environment variables are set, trigger a deploy in Railway.
+
+The build pipeline will:
+
+1. install Python dependencies
+2. install frontend dependencies in `web/`
+3. build the Vue frontend into `git_ignore_folder/static`
+4. start `rdagent server_ui`
+
+## 5. Validate the deployment
+
+After Railway finishes deploying:
+
+1. open the Railway public URL
+2. confirm the frontend loads
+3. start a task from the UI
+4. confirm traces stream normally
+5. after the task produces output, verify that artifacts appear in Supabase Storage under:
+   - `uploads/...`
+   - `traces/...`
+   - `stdout/...`
+
+## Current architecture notes
+
+- The current codebase still executes RD-Agent jobs as subprocesses inside the same `server_ui` service.
+- Supabase persistence now protects uploaded files, trace logs, and stdout artifacts against container replacement.
+- A fully split Web/API + Worker deployment would require an additional queue / orchestration layer; that is not wired in this repository yet.

nixpacks.toml

@@ -0,0 +1,16 @@
+[phases.setup]
+nixPkgs = ["python311", "nodejs_20"]
+
+[phases.install]
+cmds = [
+  "python3 -m ensurepip --upgrade",
+  "python3 -m pip install --upgrade pip setuptools wheel",
+  "python3 -m pip install .",
+  "cd web && npm ci"
+]
+
+[phases.build]
+cmds = ["cd web && npm run build:flask"]
+
+[start]
+cmd = "rdagent server_ui --port ${PORT:-19899}"

railway.json

@@ -0,0 +1,12 @@
+{
+  "$schema": "https://railway.com/railway.schema.json",
+  "build": {
+    "builder": "DOCKERFILE"
+  },
+  "deploy": {
+    "healthcheckPath": "/",
+    "healthcheckTimeout": 300,
+    "restartPolicyType": "ON_FAILURE",
+    "restartPolicyMaxRetries": 10
+  }
+}