I was going through the ICML 2026 program in Seoul and roughly classified the papers I read. The shape I kept seeing: the model is increasingly its own teacher — self-play, self-reward, self-correction and self-evolving memory move from tricks to training regimes.
Scope: self-improvement, reasoning, agents, retrieval & ranking, RL/RLHF, efficiency, generation, evaluation & theory | a curated, verified sample — not the full 6,600+ | ↗ Reading list
My overall read: the center of gravity this year is self-improvement. The interesting question stopped being “how do we supervise the model” and became “how does the model supervise itself” — and, one level up, how does it evolve the machinery (memory, verifier, reward) that does the supervising.
Roughly, the papers fell into these groups
1 Agents that rewrite themselves
The headline shift: agents that improve by editing their own experience, memory, or policy rather than by another round of human fine-tuning. Several papers push past agent-level tuning to meta-evolution — evolving the memory architecture itself, not just its contents. The sharpest open question is faithfulness: at least one paper shows these self-evolvers can drift or deceive themselves, so “it improved on the benchmark” is not yet “it improved.”
Anchors: MemEvolve: Meta-Evolution of Agent Memory Systems ICML (ICML) · Large Language Model Agents Are Not Always Faithful Self-Evolvers ICML (ICML) · From Interactions to Principles: Experience-Driven Self-Distillation for Evolving LLM Agents ICML (ICML)
2 Self-play is back, and it is aimed at code
Self-play returned as a serious training signal, now pointed at software engineering and quantitative reasoning rather than board games. The recurring recipe is a single model that both proposes a challenge (inject a bug, set a market) and solves it, with an anchor or reference set to stop the two halves drifting into an unrealistic private language.
Anchors: Toward Training Superintelligent Software Agents through Self-Play SWE-RL ICML (ICML · arXiv) · Anchoring Self-Play for Code Repair ICML (ICML) · Evolving Quantitative Reasoning through Self-Play in Digital Twin Markets ICML (ICML)
3 Reward without humans: self-reward, self-consistency, weak verification
A cluster on where the reward comes from when nobody labels anything. Some turn self-supervised objectives into dense verifiable rewards; some let the model meta-evaluate its own outputs; and a theory strand asks when a cheap self-check (self-consistency, a proxy reward) is actually good enough to trust versus when you need a strong verifier.
Anchors: Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation ICML (ICML) · SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning ICML (ICML) · When to Trust the Cheap Check: Weak and Strong Verification for Reasoning ICML (ICML)
4 Test-time RL and self-correction as first-class training
Correction moved from a decoding trick to an objective you train against. Test-time RL updates the policy from self-generated pseudo-labels at inference; multi-attempt training rewards a model for fixing its own earlier answer. The tension is reward hacking — self-generated labels are exactly the ones a model is biased to agree with.
Anchors: Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning ICML (ICML) · Learning to Correct: Reinforcement Learning for Multi-Attempt Chain-of-Thought ICML (ICML)
5 The judge problem gets quantified
As evaluation leans on LLM-as-judge, the papers stopped assuming the judge is neutral and started modelling its reliability — per-judge discriminators, judge-aware ranking, and direct measurement of judge bias. If you are going to reward a model with another model, this is the load-bearing wall.
Anchors: A Judge-Aware Ranking Framework for Evaluating LLMs without Ground Truth ICML (ICML) · Who can we trust? LLM-as-a-jury for Comparative Assessment ICML (ICML) · Quantifying Biases in LLM-as-a-Judge Evaluations ICML (ICML)
6 Retrieval learns to reason — and to act
Retrieval stopped being a fixed first stage. Reasoning-intensive retrievers and rerankers are trained to think before they rank, and deep-research agents optimise the whole search loop end-to-end with RL. The open question is cost: agentic, multi-turn retrieval is powerful but expensive, so a parallel push shrinks the encoders and prunes the reranking.
Anchors: DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research ICML (ICML · arXiv) · Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning ICML (ICML · arXiv) · ModernVBERT: Towards Smaller Visual Document Retrievers ICML (ICML · arXiv)
One quieter thread: the generative-modelling theory kept its edge
Away from the agent noise, the diffusion/flow-matching theory was some of the sharpest work — two of the Outstanding Paper awards went here: High-Accuracy Sampling for Diffusion Models and Log-Concave DistributionsICML (arXiv) and The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language ModelsICML, the latter a useful cold-shower on arbitrary-order generation.
Appendix
The full list, grouped the way I sorted it
The ICML 2026 papers I went through, each with a one-line note and its ICML program link (arXiv where I could verify a preprint). This is a curated sample, not the whole proceedings.
1 · Self-improving & self-evolving agents
ICMLSE-GA: Memory-Augmented Self-Evolution for GUI Agents — A GUI agent that pairs hierarchical memory with an iterative self-improvement pipeline (test-time memory extension plus memory-augmented self-evolution training).ICML
ICMLLarge Language Model Agents Are Not Always Faithful Self-Evolvers — Shows self-evolving agents that reuse past experience can fail to faithfully reflect it — a caution on trusting self-reported improvement.ICML
ICMLFrom Interactions to Principles: Experience-Driven Self-Distillation for Evolving LLM Agents — EvolveR distills an agent's own interaction history into reusable strategic principles and feeds them back via self-distillation.ICML
ICMLOn the Generalization Gap in Self-Evolving Language Model Reasoning — Measures how far self-generated supervision sits from oracle supervision across four self-evolution methods on controlled logical reasoning.ICML
2 · Recursive self-improvement & self-play
ICMLMemEvolve: Meta-Evolution of Agent Memory Systems — Jointly evolves an agent's experiential knowledge and its memory architecture, so the system improves how it learns, not just what it knows.ICML
ICMLToward Training Superintelligent Software Agents through Self-Play SWE-RL — One LLM trained by RL self-play to inject and repair increasingly hard bugs specified by test-suite improvements; beats a human-data baseline on SWE-bench.ICMLarXiv
ICMLAnchoring Self-Play for Code Repair — Generator-fixer self-play in a single model, anchored to a reference set so injected bugs stay close to real-world ones.ICML
ICMLEvolving Quantitative Reasoning through Self-Play in Digital Twin Markets — Agents test and adapt strategies via self-play in a controllable digital-twin market, forming a bounded self-evolving loop.ICML
3 · Self-reward, self-verification & self-consistency
ICMLConversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation — CoNL unifies generation, evaluation and meta-evaluation via multi-agent self-play, beating self-rewarding baselines without external judges.ICML
ICMLSSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning — Turns self-supervised objectives into dense, automatic, verifiable rewards for RL fine-tuning — no human preferences or AI judges.ICML
ICMLWhen to Trust the Cheap Check: Weak and Strong Verification for Reasoning — Analyzes when weak verification (self-consistency, proxy rewards) suffices versus when a strong verifier is required.ICML
ICMLOptimal Self-Consistency for Efficient Reasoning with Large Language Models — Derives power-law scaling for self-consistency and analyzes fixed- versus dynamic-allocation sampling for chain-of-thought.ICML
4 · Self-correction, self-reflection & test-time RL
ICMLBeyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning — SR-TTRL uses self-reflective verification to produce high-fidelity pseudo-labels for test-time RL instead of majority voting.ICML
ICMLLearning to Correct: Reinforcement Learning for Multi-Attempt Chain-of-Thought — Trains reasoning models to make multiple verified attempts with self-reflection, improving self-correction over long chains.ICML
ICMLTest-Time Reinforcement Learning for Flow Matching — Flow-TTRL aligns text-to-image generation on the fly by reinterpreting intermediate latents as an implicit policy.ICML
5 · Self-training & self-distillation
ICMLSelf-Distillation Enables Continual Learning — Self-Distillation Fine-Tuning learns on-policy directly from demonstrations, supporting continual learning without catastrophic forgetting.ICML
ICMLSelf-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models — OPSD has one LLM act as both teacher and student under different contexts to improve reasoning.ICML
6 · Reasoning & test-time compute
ICMLAsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding — Cuts the KV-cache bottleneck of long chain-of-thought test-time scaling with asynchronous sparse decoding, >20% faster per token at matched accuracy.ICML
ICMLUniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scaling — Casts model routing plus test-time scaling as a contextual bandit over the quality-cost trade-off.ICML
ICMLHERMES: Towards Efficient and Verifiable Mathematical Reasoning in LLMs — A tool-assisted agent interleaving informal reasoning with formally verified Lean proofs, raising accuracy while spending fewer tokens.ICML
7 · Agents, tool use & multi-agent systems
ICMLMulti²: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments — Decomposes agent behavior into complementary hierarchical roles for decision-making in interactive environments.ICML
ICMLAgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditions — Systematically evaluates tool-using agents under both user-side instruction noise and tool-side result noise.ICML
ICMLSciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents — A scalable environment with 1,780 domain tools across four natural sciences plus a tiered agentic tool-use benchmark.ICML
ICMLAppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use — 306 user-in-the-loop tasks needing diverse agent-user interaction across nine simulated apps.ICML
ICMLExCyTIn-Bench: Evaluating LLM Agents on Cyber Threat Investigation — Evaluates agents on cyber threat investigation using questions derived from investigation graphs.ICML
8 · Reasoning-first ranking & LLM rerankers
ICMLRethinking the Reranker: Boundary-Aware Evidence Selection for Robust Retrieval-Augmented Generation — BAR-RAG reframes the RAG reranker as a boundary-aware evidence selector targeting the generator's usable zone, ~10% end-to-end QA gain under noisy retrieval.arXiv
ICMLVery Efficient Listwise Multimodal Reranking for Long Documents — ZipRerank shortens input via query-image early interaction and scores all candidates in one forward pass, up to 10x lower reranking latency on MMDocIR.arXiv
ICMLRanking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domains — METEORA swaps relevance re-ranking for rationale-driven evidence selection with an elbow-detection cutoff and a verifier LLM, more accurate with fewer chunks.arXiv
9 · Agentic retrieval, deep research & RAG
ICMLDR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research — An open long-form deep-research agent trained with rubrics that co-evolve with the policy to give discriminative on-policy feedback.ICMLarXiv
ICMLDECOR: Learning to Decompose and Collaborate in Deep Search via Multi-Agent Reinforcement Learning — Multi-agent RL that decomposes deep-search tasks and coordinates collaborating agents on complex information-seeking queries.ICML
ICMLGraph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning — Builds a lightweight knowledge hypergraph and models retrieval as multi-turn agent-environment interaction, optimised end-to-end with RL.ICMLarXiv
ICMLFrom Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAG — MA-RAG turns semantic conflict among candidate answers into new retrieval queries across agentic rounds for medical QA.ICMLarXiv
10 · Dense / sparse / multi-vector retrieval
ICMLHypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation — Hyperbolic dense retrievers in Lorentz space with geometry-aware pooling, beating Euclidean baselines on MTEB and RAGBench.ICMLarXiv
ICMLAOEB: Benchmarking Agent-Oriented Multimodal Embeddings — A multi-task multimodal embedding benchmark for agent-centric retrieval (code, tool, reasoning, memory) beyond MTEB.ICML
ICMLModernVBERT: Towards Smaller Visual Document Retrievers — A 250M-parameter vision-language encoder for visual document retrieval that matches models up to 10x larger, CPU-efficient.ICMLarXiv
11 · RL, RLHF & preference optimization
ICMLMulti-Objective Preference Optimization: Improving Human Alignment of Generative Models — MOPO maximizes a primary preference objective under tunable lower bounds on secondary objectives via KL-regularized constraints.ICML
ICMLReliability-Aware LLM Alignment from Inconsistent Human Feedback — Tackles DPO's tendency to overfit ambiguous, high-disagreement preference pairs by modelling annotator reliability.ICML
ICMLAutoregressive Direct Preference Optimization — ADPO folds autoregressive structure explicitly into the DPO objective.ICML
ICMLDistributionally Robust Reinforcement Learning with Human Feedback — Distributionally robust formulations for both reward-based RLHF and reward-free DPO.ICML
ICMLAdvantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation — Diagnoses and mitigates advantage-collapse failure modes in GRPO-style RL fine-tuning.ICML
12 · Efficiency: MoE, decoding & distillation
ICMLVegas: Self-Speculative Decoding with Verification-Guided Sparse Attention — Identifies critical KV entries as a byproduct of verification and loads only those when drafting subsequent tokens.ICML
ICMLSoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs — Replaces discrete MoE routing with a truncated soft top-k relaxation, enabling gradient-based routing while activating fewer experts.ICML
ICMLReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE Inference — Router fine-tuning to raise token-wise expert reuse for fine-grained MoE under memory-constrained inference.ICML
ICMLMean Flow Distillation: Robust and Stable Distillation for Flow Matching Models — A stable distillation method for flow-matching models, SoTA on 4D occupancy forecasting and text-to-image.ICML
13 · Diffusion & flow-based generation
ICMLHigh-Accuracy Sampling for Diffusion Models and Log-Concave Distributions — Outstanding Paper: training-free first-order rejection sampling giving polylog(1/δ)-step high-accuracy diffusion sampling from scores alone.ICMLarXiv
ICMLThe Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models — Outstanding Paper questioning whether arbitrary generation order actually helps diffusion language models.ICML
ICMLA Kinetic-Energy Perspective of Flow Matching — Introduces Kinetic Path Energy, an action-like diagnostic of accumulated effort along a flow ODE trajectory.ICML
14 · Evaluation & LLM-as-judge
ICMLA Judge-Aware Ranking Framework for Evaluating LLMs without Ground Truth — Improves human-preference agreement in ground-truth-free evaluation by modelling judge reliability and calibrating ranking uncertainty.ICML
ICMLWho can we trust? LLM-as-a-jury for Comparative Assessment — A Bradley-Terry extension with per-judge discriminator parameters to jointly infer rankings and judge reliability.ICML
ICMLQuantifying Biases in LLM-as-a-Judge Evaluations — Systematically measures biases in LLM-as-a-judge autograders used as a scalable stand-in for human evaluation.ICML
ICMLQEDBench: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs — First benchmark contrasting course rubrics with expert criteria for grading undergraduate math proofs.ICML
15 · Alignment, safety & interpretability
ICMLMultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety — 10,389 adversarial prompts spanning 2,665 distinct harmful intents for multi-turn jailbreak evaluation.ICML
ICMLSteering Beyond the Support: Adversarial Training on Unsupervised Jailbroken Activation Simulation — A bi-level adversarial training scheme for zero-shot jailbreak defense, mostly <5% attack success across three LLMs.ICML
ICMLJailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking — A multi-agent system that turns published jailbreak papers into executable attack modules for reproducible safety benchmarks.ICML
ICMLLALM-as-a-Judge: Benchmarking Large Audio-Language Models for Safety Evaluation in Multi-Turn Spoken Dialogues — Benchmarks large audio-language models as safety judges in multi-turn spoken dialogue.ICML
16 · Theory: optimization & generalization
ICMLGeneralization and Scaling Laws for Mixture-of-Experts Transformers — A generalization/scaling theory for MoE Transformers that separates active per-input capacity from routing combinatorics.ICML
ICMLUnderstanding Generalization and Forgetting in In-Context Continual Learning — Extends single-task in-context-learning theory to task sequences to characterize generalization and forgetting.ICML
ICMLDynamics and representation structure of local approximations to gradient-based learning in linear recurrent neural networks — Uses dynamical-systems theory on data-aligned linear RNNs to compare stability and convergence of local gradient approximations.ICML
How I built this: I read across the ICML 2026 program and sorted a curated sample into the buckets above, weighting the self-improvement clusters because that was the year's throughline. This is not a complete pass — ICML 2026 accepted 6,600+ papers and I covered a fraction. Every paper here was checked against its icml.cc/virtual/2026 program page (which confirms the exact title and venue); ICML publishes through OpenReview/PMLR rather than assigning DOIs, so the link chip points at the program page, with arXiv added only where I could match a preprint. Nothing here is a guessed link.