Robust In-Context Reinforcement Learning Under Reward Poisoning Attacks

Paulius Sasnauskas^†,1,2, Yiğit Yalın³, Goran Radanović³

^†This work was done as a part of an internship project at MPI-SWS.
¹Department of Computing Science, University of Alberta, Edmonton, Canada.
²Alberta Machine Intelligence Institute (Amii), Edmonton, Canada.
³MPI-SWS, Saarbrücken, Germany

PDF: arxiv.org/pdf/2506.06891

Published at ICML 2026. Code: github.com/PauliusSasnauskas/AT-DPT

Conference page: icml.cc/virtual/2026/poster/61251

TL;DR

In-context RL agents like the Decision-Pretrained Transformer (DPT) learn a learning algorithm from the rewards in their context – which means an attacker can steer them at test time just by poisoning those rewards. We introduce AT-DPT (Adversarially-Trained DPT), which trains a population of attackers to corrupt rewards while the agent learns to recover. AT-DPT stays robust across a wide range of attacks – including ones it never saw during training – and substantially outperforms standard and corruption-robust bandit/RL baselines.

Abstract

We study the corruption-robustness of in-context reinforcement learning (ICRL), focusing on the Decision-Pretrained Transformer (Lee et al., 2023). To address the challenge of reward poisoning attacks targeting the DPT, we propose a novel adversarial training framework, called Adversarially Trained DPT (AT-DPT). Our method simultaneously trains a population of attackers to minimize the true reward of the DPT by poisoning environment rewards, and a DPT model to infer optimal actions from the poisoned data. We evaluate the effectiveness of our approach against standard bandit algorithms, including robust baselines designed to handle reward contamination. Our results show that AT-DPT significantly outperforms them in bandit settings under a learned attacker, and generalizes to more complex environments such as adaptive attackers and MDPs. It shows promise in ICRL as a meta-RL approach to learning effective corruption-robust algorithms.

Problem: Poisoning the Learning Algorithm, Not the Policy

Most prior work on reward poisoning attacks targets the canonical RL setting: an attacker corrupts rewards during training to change the fixed policy an agent eventually adopts. These are training-time attacks, and they assume the agent's test-time behavior no longer depends on rewards.

In-context RL breaks that assumption. An ICRL agent implements a learning algorithm in-context: its context encodes past interactions, including rewards, and it adapts its behavior on the fly. So an adversary who corrupts rewards at test time can still influence the agent – effectively attacking the learning algorithm running inside the transformer's context. To our knowledge, this is the first study of reward poisoning in meta-RL.

Attack Model

We use Huber's ε-contamination model: at each timestep, with probability ε the agent observes a corrupted reward produced by the attacker instead of the true reward. The attacker sees the true reward (and, if adaptive, the agent's recent interactions) and aims to minimize the agent's true return under a soft budget constraint (a penalty term, with budget B). We focus on non-behavior-targeted attacks – the attacker wants to hurt return, not force a specific policy.

Method: AT-DPT

AT-DPT combines in-context learning with adversarial training. Building on DPT (a GPT-2-style transformer trained to predict optimal actions from context), we add an adversarial training stage between pretraining and deployment. We frame robust ICRL as approximately solving a bi-level game and search for a Nash equilibrium between the agent and a per-task attacker.

Training proceeds in three phases:

1. Pretraining. A GPT-2 model is trained by supervised learning to predict an oracle's optimal action from a context dataset and a query state (the original DPT objective).
2. Adversarial training. For each of M sampled tasks, a dedicated attacker is trained with REINFORCE to minimize the agent's true return under the budget penalty, while the agent is simultaneously trained to predict optimal actions from the poisoned context. The agent sees corrupted rewards; the attacker sees clean rewards. This repeats for N rounds.
3. In-context deployment. The frozen agent is deployed in a new, corrupted environment. Unlike the original DPT, context is updated after every timestep (not just per-episode), which supports robustness against adaptive attackers.

We consider direct-parameterized non-adaptive attackers (per-arm or per-state-action reward shifts) and adaptive attackers that use the same transformer architecture as the victim to react to its defenses.

Results

We evaluate on multi-armed bandits, linear bandits, and MDPs (Darkroom2 and the 3D Miniworld navigation environment), comparing against standard and corruption-robust baselines. To avoid testing on the exact attack seen during training, attackers are cross-validated across seeds. All numbers are mean ± 95% confidence interval (2×SEM) over 10 replications, at ε = 0.4.

Bandit Setting – Cumulative Regret (lower is better)

Each column is the algorithm the attacker was trained against (the "attacker target"). AT-DPT keeps low regret no matter which method the attack was tuned for – including attacks targeting other algorithms – while the baselines blow up under a learned attack. Budget B = 3.

*We use tuned versions of RTS and crUCB which outperform base versions

Linear Bandit Setting – Cumulative Regret (lower is better)

Here the gap is dramatic: under a learned attack AT-DPT reaches regret around 2–3 while every baseline sits at 35–70 – an improvement of more than 90% – and AT-DPT still matches the corruption-robust CRLinUCB in the clean case. Budget B = 3.

MDP Setting – Darkroom2, Average Episode Reward (higher is better)

In a sparse-reward 2D gridworld with two goals, AT-DPT stays robust across attackers, with the task-specific NPG coming closest. Note that NPG and Q-learning are run for 100 episodes per task before evaluation, whereas AT-DPT infers the task in-context almost immediately. Budget B = 10.

^§NPG and Q-learning need multiple episodes of learning to converge to a stable policy; we run them for 100 episodes before evaluating performance.

AT-DPT also displays similar results in the 3D Miniworld visual-navigation environment, where it again stays more robust than DPT and a trained PPO agent.

Why It Works

• Informed prior over tasks. Because AT-DPT is a meta-learner over a task distribution, corruption schemes that are implausible under that prior simply don't fool it.
• Corrected posterior sampling. AT-DPT can be viewed as in-context posterior sampling with a posterior that accounts for the corruption process. As long as the true reward means stay identifiable under corruption, standard posterior-sampling regret analyses apply.
• Generalization across attacks. Training against one attacker yields robustness against unseen attacks, including adaptive ones – echoing the PGD-robustness principle from adversarial training in supervised learning.

Why It Matters

Robustness to test-time poisoning is increasingly important for any agent that learns in-context. A prime example is LLM-based agents, whose context can be poisoned through external tools or retrieval (RAG) – and LLMs have themselves been shown to exhibit ICRL behavior. By learning a learning algorithm that is inherently robust and that exploits prior knowledge of the task distribution, AT-DPT points toward more efficient, corruption-robust decision-making agents.

Limitations and Future Work

• Like DPT, training needs optimal actions from an oracle. Suboptimal demonstrations are tolerated to a point (the "sub." rows above), but performance degrades as their fraction grows. Training on offline trajectories with simulated attackers is a promising alternative.
• Each experiment uses a single attack specification. Mixing contamination levels and attack types (adaptive and non-adaptive together) during training could improve robustness further and soften the small clean-environment trade-off.
• A full theoretical account – proving convergence to an approximate equilibrium of the underlying game – is left to future work. Extensions to other ICRL methods (e.g. Vintix, ICPE) are also open.

Acknowledgements

This research was, in part, funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number 467367360. We acknowledge the use of computing resources generously provided by MPI-SWS.