Add gangscheduling demos

README.md

@@ -9,18 +9,13 @@ This project aims to get up to speed with AIchor by going through the whole proc
 
 ## How to use it ?
 
-You can find multiple manifests samples in the `manifests` directories. If you want to try hugging face accelerate for example, all you need to do is to copy it:
+You can find multiple manifests samples in the `aichor_manifests` directories. If you want to try hugging face accelerate for example, all you need to do is to copy it:
 
 ```bash
-$ cp hugging-face-accelerate/manifests/single_worker/manifest.1-wrkr-1-a100-80gb.yaml manifest.yaml
 
-# also works with
-# cp smoke-test/manifests/manifest.kuberay.sample.yaml manifest.yaml
-# cp smoke-test/manifests/manifest.pytorch.sample.yaml manifest.yaml
-# cp parallel-jobs-demo/manifests/manifest.yaml manifest.yaml
+$ git add aichor_manifests/hugging-face-accelerate/rdma/1_worker/manifest.1wrkr-1-h100-80gb.yaml
+$ git commit -m "aichor[hugging-face-accelerate/rdma/1_worker/demo-manifest.yaml]: experiment" # commit has "aichor[<manifest-path>]: " to trigger experiment where `<manifest-path>` is the path of manifest relative to the `aichor_manifests/` directory
 
-$ git add manifest.yaml
-$ git commit -m "exp: eriment" # commit has to start by "exp: " to trigger experiment
 $ git push
 ```
 

gangscheduling-demo/README.md

@@ -0,0 +1,57 @@
+# Gang scheduling
+
+In here we will go through what gang scheduling is, what it does, and when we should (and shouldn't) use it, using demos across the different frameworks and operators supported by AIchor: Ray (kuberay), JAX, PyTorch, and a plain JobSet.
+
+## What is gang scheduling?
+
+Distributed training jobs are made up of several pods that only make progress once **all** of them are running together (e.g. a PyTorch job's workers all need to rendezvous, a Ray cluster needs its head and workers up, a JAX SPMD job needs every host present before the collective ops can run).
+
+The default Kubernetes scheduler places pods one at a time, with no notion that a group of pods belongs together. On a busy, resource-constrained cluster this can lead to:
+
+- **Partial deployment / deadlock**: half of a job's pods get scheduled and start consuming resources, while the rest sit pending because there isn't room left. The job can't make progress, but it's also not releasing what it already holds, which can block other jobs from ever getting enough resources either.
+- **Wasted compute**: pods that did get scheduled sit idle waiting for the rest of the group, burning GPU/CPU allocation without doing useful work.
+
+**Gang scheduling** (a.k.a. co-scheduling) fixes this by scheduling a group ("gang") of pods as an atomic unit: either enough resources exist to place the whole group and they all start together, or none of them are scheduled and the job waits as a whole until it can.
+
+## It's about coupling, not pod count
+
+Having multiple replicas in a job does **not** automatically mean it needs gang scheduling. What matters is whether those replicas are **tightly coupled** (they must all be present at once to make any progress) or **independent** (each replica can do useful work on its own, regardless of whether its siblings are scheduled yet). Every framework here can be used either way:
+
+- **JobSet**
+  - Tightly coupled: replicas are workers of a single distributed job and need each other to make progress — needs gang scheduling.
+  - Independent: [`parallel-jobs-demo`](../parallel-jobs-demo) in this repo runs a JobSet where each replica is a self-contained job (e.g. one arm of a hyperparameter sweep) with no rendezvous between replicas — doesn't need gang scheduling. Replicas can be scheduled as capacity frees up, so the cluster makes partial progress instead of waiting for room for all of them at once.
+
+- **PyTorch**
+  - Tightly coupled: a DDP/FSDP job where every worker must rendezvous before training can start stepping — needs gang scheduling.
+  - Independent: several separate PyTorch training runs submitted as replicas of the same job (e.g. training multiple model variants, or a sweep), each running its own single-process training loop with no dependency on the others — doesn't need gang scheduling.
+
+- **JAX**
+  - Tightly coupled: a multi-host SPMD job where every host must be present before collective ops can execute — needs gang scheduling.
+  - Independent: several separate single-host JAX programs run as replicas (e.g. a parameter sweep, each on its own devices) — doesn't need gang scheduling, since there's no cross-replica rendezvous.
+
+- **Ray (kuberay)**
+  - Tightly coupled: cluster bootstrap — the head node and whatever minimum worker count a job actually needs to start must come up together — needs gang scheduling for that initial set.
+  - Independent: once the cluster is up and autoscaling, additional workers requested as load ramps up don't need to arrive together — Ray puts each new worker to use as soon as it lands. Gang scheduling those incremental requests would work against the point of autoscaling.
+
+## When to use it
+
+- You're running **tightly-coupled multi-pod jobs** — replicas that must rendezvous to make progress, such as PyTorch DDP/FSDP, JAX multi-host SPMD, a Ray cluster's initial head + minimum-worker bootstrap, or a JobSet whose replicas depend on each other.
+
+## When it's not relevant
+
+- **Independent-replica workloads**, such as the parallel job fan-out in [`parallel-jobs-demo`](../parallel-jobs-demo), a sweep of standalone PyTorch/JAX runs, or an autoscaling Ray cluster's incremental worker additions — each replica is useful on its own, so all-or-nothing scheduling only adds latency for no benefit.
+- Single-pod jobs (e.g. a single-worker training script) have nothing to "gang" — there's only one pod to schedule.
+
+## Demos
+
+Each subfolder is a minimal, self-contained AIchor project showing gang scheduling and independent scheduling (parallel scheduling) with a different operator:
+
+| Demo | Operator | What it shows |
+| --- | --- | --- |
+| [`jobset-gangscheduling-demo`](./jobset-gangscheduling-demo) | `jobset` | Multi-replica JobSet whose replicas must rendezvous — the tightly-coupled case, contrast with the independent-replica setup in [`parallel-jobs-demo`](../parallel-jobs-demo). |
+| [`pytorch-gangscheduling-demo`](./pytorch-gangscheduling-demo) | `pytorch` | Multi-worker PyTorch distributed job. |
+| [`jax-gangscheduling-demo`](./jax-gangscheduling-demo) | `jax` | Multi-host JAX SPMD job. |
+| [`kuberay-gangscheduling-demo`](./kuberay-gangscheduling-demo) | `kuberay` | Ray cluster where the head + minimum worker set is gang-scheduled to bootstrap, while additional autoscaled workers join independently as demand ramps up. |
+
+
+> More detail on each operator's specific setup lives in that demo's own README.