Learning Self-Orchestration in Decentralized Agent Swarms

Abstract

The capabilities of an agent swarm depend on both its individual members and how they coordinate. Yet existing multi-agent systems often train agents within fixed orchestration structures, separating the learning of individual capabilities from the organization of collective behavior. We introduce market-based post-training (MPT), a framework for jointly learning action policies and self-orchestration in decentralized agent swarms. Agents express their willingness to participate through bids, which determine who acts and how collective outcomes provide local training credit. Learning both action and bidding behavior allows individual capabilities and patterns of participation to adapt together without a central coordinating agent. We evaluate MPT on embodied crafting, robot navigation, and deep research. Results show gains over monolithic post-training and confirm that agent participation indeed evolves with learning. With the same post-trained action policies, learned self-orchestration outperforms alternative coordination strategies. Ablations show that both action and bidding benefit from learning, and that learned bids provide more effective action-training signals. Beyond the original training objectives, trained swarms adapt faster to related tasks and achieve better performance with larger populations at inference time. Larger populations also suffer smaller performance losses under agent dropout, demonstrating that learned self-orchestration supports reuse, scaling, and robustness in decentralized agent swarms. Code and trained models are available at https://github.com/Embodied-Minds-Lab/MPT.

Benchmarks

We evaluate MPT on embodied crafting, robot navigation, and deep research. Each setting uses three agents with complementary roles and independent model parameters.

Method

Learning to Act

At the end of each episode, local training credit is assigned by propagating rewards backward through the sequence of acting agents. With \(\hat{r}_t\) the environment reward obtained at turn \(t\) (the terminal verified reward is included in \(\hat{r}_T\)), the accumulated value \(v_t\) and the local training credit \(\tilde{r}_t\) are

\[ v_t = \hat{r}_t + b_{t+1} v_{t+1}, \qquad \tilde{r}_t = (1 - b_t) v_t, \qquad v_{T+1} = 0. \]

The acting agent retains a fraction \(1 - b_t\) for its own action update and passes the remaining fraction backward; the share passed backward by the first acting agent is discarded. This allows downstream outcomes to provide learning signals for earlier actions, following the principle of a bucket brigade. Each agent updates its own parameters using only the action tokens from turns in which it acted, with REINFORCE and a per-agent mean baseline:

\[ \nabla_{\theta_i} J = \mathbb{E}_{t \in \mathcal{W}_i} \left[ (\tilde{r}_t - \bar{r}_i) \nabla_{\theta_i} \log \pi_i(a_t \mid h_{i,t}) \right], \]

where \(\mathcal{W}_i\) contains the turns assigned to agent \(i\) and \(\bar{r}_i\) is its mean training credit over those turns in the batch.

Learning to Self-Orchestrate

Bids are learned from counterfactual comparisons. The original branch is the collected trajectory, in which the selected agent acts at the contested turn. To construct the alternative branch, the environment and agent states at the contested turn are restored in a replica and the selected agent's bid is resampled until it falls below the runner-up's bid, so that the runner-up acts instead. The alternative branch is continued until the episode terminates, and each branch is scored by its sum of environment rewards from the contested turn onward. The bidding response of the higher-scoring branch becomes the preferred response \(y_w\) and the other the rejected response \(y_l\). The agent is trained on these pairs with a Bradley–Terry preference objective:

\[ \mathcal{L}_{\text{bid}} = -\log \sigma(\log \pi_i(y_w \mid h_{i,t}) - \log \pi_i(y_l \mid h_{i,t})). \]

Experiments

Main Results

MPT Improves Performance at Matched Training Compute

We compare MPT with monolithic RL under matched training FLOP budgets across the three tasks.

Held-out Evaluation

We report results on held-out sets disjoint from training: 32 unseen world seeds for embodied crafting, 88 R2R-CE val-unseen episodes for robot navigation, and 50 held-out OpenResearcher questions for deep research. Each instance is evaluated three times with the training settings. MPT outperforms monolithic RL on all three tasks.

Analysis

Self-Orchestration Evolves during Training

To examine how self-orchestration develops through learning, we track agent participation on the Iron Pickaxe task across four checkpoints: after SFT but before RL, at two intermediate stages of RL, and after RL. Before RL, the Logger dominates throughout the episode, while the Miner and Smith participate infrequently. As training progresses, the Logger remains predominant at early steps, but its share decreases later as the Miner and Smith become more involved. At decision step 120, the Logger's participation share decreases from approximately 80% before RL to 45% after RL, while the Miner's share increases from approximately 10% to 40% and the Smith's from approximately 10% to 15%.

Comparison between Self-Orchestration and Other Coordination Strategies

We evaluate whether learned self-orchestration coordinates agents more effectively than alternative inference-time strategies. Keeping the post-trained action policies fixed, we compare our bidding mechanism with Round-Robin, AgentNet, and Graph-of-Agents (GoA) on Iron Pickaxe over the same 32 seeds. MPT achieves a mean episode reward of 41.5, compared with 25.4 for GoA, the strongest alternative coordination strategy, a 63.4% relative improvement.

Both Bidding and Action Benefit from Learning

To check that both components benefit from learning, we also use either the initial or the post-trained checkpoint for bidding and for action independently: training both yields 2.69 times the reward of the initial configuration.

Isolating the Contributions of Learned Self-Orchestration

To isolate the contribution of learned self-orchestration, we conduct two ablations on the Iron Pickaxe task. Both ablations share the agent roles, SFT initialization, and action-training procedure of MPT, and each modifies a single component. Round-robin allocates turns cyclically and trains on the reward with \(\gamma = 0.95\), removing bidding entirely. Random credit replaces the bids used for credit assignment with values sampled from \(\mathcal{U}(0, 1)\), leaving turn allocation and bidding optimization unchanged. At approximately \(5 \times 10^{18}\) training FLOPs, MPT reaches a mean episode reward of about 42, compared with 15 for Round-robin and 31 for random credit.

Reusability, Scalability, and Robustness

Reusing Learned Self-Orchestration

To support a growing range of tasks, agent swarms need to build on previously learned skills and coordination. Starting from their Iron Pickaxe checkpoints, we continue training MPT and monolithic RL separately on two additional crafting tasks, Torch and Stone Bricks. At approximately \(10^{18}\) adaptation FLOPs, the society achieves mean episode rewards of about 35 on Torch and 38 on Stone Bricks, compared with 17 and 19 for the monolithic baseline.

Larger Swarms Improve Inference-Time Performance

Decentralized agent swarms should scale at inference time. We vary the population size across 3, 6, 9, and 12 agents by using one to four copies of each trained agent, on Iron Pickaxe over 32 seeds with three runs per seed, and evaluate pass@3 on five subtasks. Increasing the population from 3 to 12 agents raises pass@3 from approximately 78% to 97% on Wooden Pickaxe, from 69% to 94% on Stone Pickaxe, and from 66% to 91% on Furnace.

Larger Swarms Better Tolerate Agent Dropout

Decentralized swarms do not rely on a central agent, which can limit the impact of occasional agent unavailability. We introduce agent dropout at inference time on Iron Pickaxe over 32 seeds, averaging three runs per seed. At a dropout probability of \(p = 0.9\), the 12-agent population maintains a mean episode reward of approximately 33, compared with 19 for the 3-agent population: additional agents provide redundancy that helps the swarm maintain its capabilities when members become unavailable.

BibTeX

If you find MPT and our prior work EoM useful in your research, please consider citing our papers.

@misc{xu2026learningselforchestrationdecentralizedagent,
      title={Learning Self-Orchestration in Decentralized Agent Swarms}, 
      author={Guowei Xu and Zhenting Qi and Amani Kiruga and Himabindu Lakkaraju and Yilun Du},
      year={2026},
      eprint={2610.22322},
      archivePrefix={arXiv},
      primaryClass={cs.MA},
      url={https://arxiv.org/abs/2610.22322}, 
}

@inproceedings{qi2026economy,
  title     = {Economy of Minds: Emerging Multi-Agent Intelligence with Economic Interactions},
  author    = {Zhenting Qi and Huangyuan Su and Ao Qu and Chenyu Wang and Yu Yao and Han Zheng and Kushal Chattopadhyay and Guowei Xu and Zihan Wang and Weirui Ye and Vijay Janapa Reddi and Ju Li and Paul Pu Liang and Himabindu Lakkaraju and Sham M. Kakade and Yilun Du},
  booktitle = {Workshop on Scientific Understanding of Foundation Models},
  year      = {2026},
  url       = {https://openreview.net/forum?id=Oznkw4Uncb}
}