待翻译：Scheduling jobs across Slurm clusters (and K8s, and cloud) from one place

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Slurm dominates large-scale computing - 60% of Top500 supercomputers rely on it for workload management. If you’re working in ML research or HPC, having access to multiple GPU clusters sounds like a good problem to have. More resources = more compute, which means faster experiments and shorter iteration cycles. In practice, however, managing multiple Slurm clusters is anything but enjoyable, it’s a manual, error-prone process that becomes a bottleneck as your team scales. Imagine this: your primary cluster is full, so you SSH to a different login node, remember which partition names are different there, tweak your job script, and resubmit. You rinse and repeat across three, four, or five different clusters. You start thinking “there’s gotta be a better way.” What if you could treat all your Slurm clusters as a single resource pool? What if your jobs could automatically find and run on whichever cluster has available GPUs? That’s exactly what SkyPilot enables. The multi-cluster pain points # So what makes managing multiple Slurm clusters difficult? No unified view of resources Each Slurm cluster is self-contained. To see what GPUs are available across your clusters, you need to: SSH to cluster-a’s login node Run sinfo or check the queue SSH to cluster-b’s login node Run sinfo again Compare in your head (or in a spreadsheet) Decide where to submit This becomes especially time consuming when you’re looking for specific GPU types (e.g. H200s vs H100 vs B300s) or need multi-node allocations. There’s no single command that shows you “here’s what’s available everywhere.” Manual failover when clusters are full When your primary cluster hits capacity, you have to: Notice that your job is stuck in the queue Realize you should try another cluster SSH to a different login node Adapt your submission script (different partition names, different module environments, different paths) Submit again Hope that cluster has capacity For teams running continuous training pipelines or parameter sweeps, this manual intervention breaks the workflow. You can’t just submit 100 jobs and let them find capacity-you have to babysit the process. Inconsistent environments across clusters Different Slurm clusters often have: Different partition naming schemes (gpu vs b300 vs h100-cluster) Different module environments (module load cuda/12.1 vs module load cuda/12.4) Different storage mounts (home directory paths, scratch filesystem locations) Different networking configurations Every time you move a job to a different cluster, you’re effectively debugging environment differences. This is both tedious and it becomes a source of subtle bugs when jobs fail due to environment mismatches. Bad observability = low utilization Without visibility across clusters, you can’t make smart scheduling decisions: Should you wait for H100s on cluster-a, or use available B300s on cluster-b? Should you preempt lower-priority jobs, or fail over to another cluster? Which cluster will give you the fastest queue time? You’ll end up underutilizing your infra based on incomplete information. What about native Slurm multi-cluster support? Slurm does offer multi-cluster operation and federation, but both have limitations. Multi-cluster mode routes jobs to the cluster with the earliest start time at submission, but “Slurm makes no subsequent effort to migrate a job to a different cluster” after that. Federation provides peer-to-peer scheduling with replicated “sibling jobs,” but the docs note it’s “not intended as a high-throughput environment” and suggest configuring fewer clusters if you’re scheduling more than 50,000 jobs a day. It also requires that all compute nodes be reachable from all submission hosts, which is a tall order for clusters in separate data centers. SkyPilot’s unified interface for Slurm # SkyPilot is a control plane that turns fragmented compute into one unified AI compute pool. While SkyPilot supports (single or multiple) Kubernetes really well, it also has first-class support for Slurm clusters, including the ability to manage multiple Slurm clusters as a unified resource pool. With SkyPilot, you get: Unified resource view: See GPUs across all your clusters with a single command-no more SSH-ing to different login nodes to check availability. Automatic cluster selection and failover: Submit a job and SkyPilot finds a cluster with capacity. If that cluster is full, it tries the next one automatically. Consistent environments: Write your task definition once. The same YAML works on any Slurm cluster (and even on cloud VMs or Kubernetes). Unified job management: Monitor, log, and cancel jobs across all clusters from one place. Cloud overflow: When all your Slurm clusters are saturated, automatically burst to cloud VMs for additional capacity. The rest of this post walks through setup, usage, and real-world workflows. Getting started # Prerequisites You need SSH access to your Slurm clusters. SkyPilot uses SSH to connect to login nodes and submit jobs via sbatch, just like you would manually. Configuration Configure your clusters in ~/.slurm/config using SSH config format: This follows the standard SSH config format, so if you’re already using SSH configs for these clusters, you can reuse them. Verify setup Check that SkyPilot can see all your clusters: You can also test connectivity to a specific cluster: SkyPilot automatically discovers available clusters from your config and tests connectivity. Key features # Unified resource view The SkyPilot dashboard provides a unified view of all your GPU clusters: SkyPilot Dashboard provides a unified view of all your slurm clusters, and shows detailed information about node health and utilization. If those who prefer the CLI, see GPUs across all your clusters with one command: This gives you instant visibility into what’s available where. No more SSH-ing to different login nodes to check queue status. Automatic cluster selection and failover Write your task once: Note: The GPU name (e.g., H100, B300, L4) must match what’s configured in your Slurm cluster’s GRES settings. SkyPilot converts this to the appropriate --gres=gpu:H100:8 directive when submitting to Slurm. Launch it: SkyPilot will: Check resource availability across all configured Slurm clusters Select the cluster with available H100s Submit the job via sbatch If the first cluster is full, automatically try the next one Handle environment setup consistently across clusters You don’t have to specify which cluster to use - SkyPilot finds capacity and schedules there. Consistent environment across clusters SkyPilot abstracts away cluster-specific details: Partition names: Specifying partitions is optional; SkyPilot can automatically map GPU requests to the right partitions on each cluster Module environments: Your setup block runs on every cluster, so dependencies are consistent Environment variables: SkyPilot provides standardized variables that work everywhere Your task definition is portable. The same YAML works on any Slurm cluster (and even on cloud VMs or Kubernetes if needed). Environment variables SkyPilot sets these environment variables automatically for multi-node jobs: SkyPilot Variable Slurm Equivalent Description $SKYPILOT_NODE_IPS $SLURM_JOB_NODELIST Resolved, newline-separated node IPs (SLURM_JOB_NODELIST contains node names) $SKYPILOT_NUM_NODES $SLURM_NNODES Total number of nodes $SKYPILOT_NODE_RANK $SLURM_PROCID Node rank (0 to N-1) $SKYPILOT_NUM_GPUS_PER_NODE $SLURM_GPUS_PER_NODE GPUs per node $SKYPILOT_TASK_ID $SLURM_JOB_ID Unique task identifier These variables make your code portable-the same script works on Slurm, cloud VMs, and Kubernetes. Interactive development across clusters Use salloc-style interactive sessions that can land on any cluster: SkyPilot finds an available H100 across all your clusters and gives you a dev cluster. SSH in: You’re now on a GPU node on whichever cluster had capacity. You don’t need to know which one-the environment is consistent. You can even set up VSCode remote development with SkyPilot dev clusters (see instructions here). Job management across clusters See all your jobs across all clusters: Cancel or stop jobs uniformly: No need to remember which cluster each job is on or SSH to different login nodes to cancel jobs. Command reference Here’s a quick reference for translating Slurm commands to SkyPilot: Task Slurm SkyPilot Interactive allocation salloc --gpus=8 sky launch -c dev --gpus H100:8 Run command srun python train.py sky exec dev python train.py Submit batch job sbatch script.sh sky jobs launch task.yaml View jobs squeue sky status or sky jobs queue Cancel job or allocation scancel sky down or sky jobs cancel View resources sinfo sky gpus list --infra slurm View logs tail -f job.log sky logs The key difference: with SkyPilot, you don’t need to specify which cluster to use. SkyPilot automatically selects the best available cluster based on resource availability. Real-world workflow # Let’s walk through a typical workflow: training multiple model variants in parallel. Define your training task Launch multiple variants SkyPilot automatically distributes these across your clusters based on availability. Maybe train-7b lands on slurm-cluster-a, train-13b on slurm-cluster-b, and train-70b on slurm-cluster-c-all without you specifying or caring. Monitor progress Check logs from any cluster No SSH-ing to the right login node-SkyPilot handles it. Advanced features # Specifying cluster preferences If you want to target a specific cluster (due to data locality, cost, etc.), specify it in the infra field: Or use the --infra flag at launch time: When you specify a cluster, SkyPilot will only use that cluster. To enable automatic failover across multiple clusters, omit the cluster specification and let SkyPilot choose based on availability. Using cloud as overflow SkyPilot supports cloud VMs alongside Slurm clusters. If all your Slurm clusters are full, you can automatically overflow to cloud: This gives you infinite overflow capacity when on-prem resources are saturated. Managed jobs for long-running training Use SkyPilot’s managed jobs feature for multi-day training runs that need fault tolerance: Managed jobs automatically: Recover from node failures Resume from checkpoints This works across all your Slurm clusters, with automatic failover if a cluster goes down. Important notes and limitations # While SkyPilot provides powerful multi-cluster management for Slurm, there are some limitations to be aware of: What’s supported: SSH-based access to existing Slurm clusters Multi-node distributed jobs Interactive development with SSH access Managed jobs with fault tolerance Automatic failover across clusters Container images via Pyxis and enroot (see Containers) Current limitations: Autostop: Not supported on Slurm clusters (you need to manually stop clusters) SkyServe: Model serving deployments are not available on Slurm Cluster provisioning: SkyPilot cannot create new Slurm clusters (only manages existing ones) Authentication & permissions: Jobs submit under your configured SSH username and respect your existing Slurm account permissions. If your account has restrictions on certain partitions or GPU types, those same restrictions apply when launching through SkyPilot. If your team runs a shared SkyPilot API server, you can enable submit as user so jobs land under each person’s own Unix account rather than a single shared login. Shared filesystems: SkyPilot automatically leverages shared filesystems (typically NFS) that are mounted on your Slurm clusters. Your home directory and any shared scratch spaces are accessible across all compute nodes without additional configuration. Storage considerations # Shared filesystems If your clusters have shared storage (e.g., [truncated for AI cost control]