What is this?
It's a complete, public recipe for RL-training knowledge-work agents on Mercor's APEX-Agents data, built on the open-source SkyRL training library and the open-source harbor sandbox/grading toolkit. It's the companion code for Mercor's 'Training Frontier Knowledge Work Agents' blogpost.
It's a complete, public recipe for RL-training knowledge-work agents on Mercor's APEX-Agents data, built on the open-source SkyRL training library and the open-source harbor sandbox/grading toolkit. It's the companion code for Mercor's 'Training Frontier Knowledge Work Agents' blogpost.Is this a reusable framework, or a specific one-off training recipe?It's a specific recipe, not a general framework: SkyRL and harbor are pip-installed as-is with zero forks, and this repo only adds the glue code (the generator, the agent, trial configs, entrypoints) around them.
It's a specific recipe, not a general framework: SkyRL and harbor are pip-installed as-is with zero forks, and this repo only adds the glue code (the generator, the agent, trial configs, entrypoints) around them.What's the actual workflow to run a training job?You first download and extract the HuggingFace task dataset, then run a 1-GPU smoke test of the full trial-to-checkpoint loop, then scale to fully-async training on 2+ GPUs, and finally run eval.
You first download and extract the HuggingFace task dataset, then run a 1-GPU smoke test of the full trial-to-checkpoint loop, then scale to fully-async training on 2+ GPUs, and finally run eval.What do I need installed before running this?You need the pinned releases from pyproject.toml: harbor[modal]==0.21.0 from PyPI and an editable skyrl[megatron] 0.3.0 checkout (it can't be pulled as a plain PyPI package because of its custom GPU wheel sources), plus access to a Ray GPU cluster, Modal, and AWS ECR.
You need the pinned releases from pyproject.toml: harbor[modal]==0.21.0 from PyPI and an editable skyrl[megatron] 0.3.0 checkout (it can't be pulled as a plain PyPI package because of its custom GPU wheel sources), plus access to a Ray GPU cluster, Modal, and AWS ECR.Can I get the actual training data myself?No — the APEX-Agents task dataset itself is not released due to licensing, only its format is documented so you can understand the architecture.
No — the APEX-Agents task dataset itself is not released due to licensing, only its format is documented so you can understand the architecture.How many GPUs does training actually need?A smoke test of the full trial loop works on a single GPU, but real fully-async training requires at least 2 GPUs on a Ray cluster.
A smoke test of the full trial loop works on a single GPU, but real fully-async training requires at least 2 GPUs on a Ray cluster.What models were trained with this recipe?Mercor published checkpoints for two models trained with this recipe: 'Qwen3.6-35B-A3B-Mercor' and 'Qwen3.5-397B-A17B-Mercor', both on HuggingFace.
Mercor published checkpoints for two models trained with this recipe: 'Qwen3.6-35B-A3B-Mercor' and 'Qwen3.5-397B-A17B-Mercor', both on HuggingFace.What do SkyRL and harbor each handle?SkyRL runs the fully-async training loop, the vLLM inference engines, and in-flight NCCL weight sync, while harbor drives each trial's rollout lifecycle (env start, agent.run(), verify, teardown) inside a Modal sandbox.
SkyRL runs the fully-async training loop, the vLLM inference engines, and in-flight NCCL weight sync, while harbor drives each trial's rollout lifecycle (env start, agent.run(), verify, teardown) inside a Modal sandbox.How are agent rollouts graded?After the agent finishes, harbor runs a grading script inside the sandbox that uses an LLM judge (a Gemini model by default) to score the output against the task's verifiers.json; the per-criterion scores become the trial's reward.
After the agent finishes, harbor runs a grading script inside the sandbox that uses an LLM judge (a Gemini model by default) to score the output against the task's verifiers.json; the per-criterion scores become the trial's reward.What are 'worlds' in APEX-Agents tasks?Worlds are simulated companies with their own documents, apps, and MCP tool servers; many tasks share one world, so environments and their ECR Docker images are built per world rather than per task.
Worlds are simulated companies with their own documents, apps, and MCP tool servers; many tasks share one world, so environments and their ECR Docker images are built per world rather than per task.Which topics is it in?agent-sandbox, llm-training, distributed-training, training, aws, dataset, vllm, reinforcement-learning, qwen, evaluation, sandbox, large-language-models
agent-sandbox, llm-training, distributed-training, training, aws, dataset, vllm, reinforcement-learning, qwen, evaluation, sandbox, large-language-models