We want to test a failure mode in long-horizon financial workflows: an agent continues a prescribed analysis after its assumptions no longer hold. For example, an agent asked to build a leveraged buyout (LBO) model may continue calculating returns after learning that the assumed borrowing is unavailable. The evaluation needs to distinguish completion of the model from a supported decision about how to proceed.
In Funding trader, the next action depends on current quotes, client constraints, approvals and available resources. The agent must identify which conditions are established, which remain unresolved and whether the proposed action is permitted at that checkpoint.
We want to evaluate decisions that account for P&L, the client's funding needs and the bank's financial, legal, regulatory and reputational risks. The current task implements a subset of that objective, with its scoring scope set out below.
Funding trader represents the task as a synthetic book with successive decision checkpoints. Accepted actions update cash, positions and obligations and produce dated receipts. The evaluator uses those records when checking later actions and reported client outcomes.
How the process becomes the objective
Training examples and rewards that recognise only one route may encourage the agent to repeat it when circumstances differ. The following examples describe the failure modes we intend to test.
Continuing after a premise fails
Consider an agent assessing a leveraged buyout. It builds a model, projects returns and recommends a price. Then it learns that the borrowing cannot be approved on the assumed terms. The useful next step is to establish which financing is available and revisit the transaction. Refining the original model leaves the decision unresolved. The recommendation remains conditional until the agent establishes which financing terms are available.
Treating an unknown as permission
An agent can fill a missing financing term or approval with an assumption so that the workflow can continue. Assumptions help explore possibilities; they cannot establish that cash or permission is available. A conditional result must stay conditional until that dependency is resolved.
Making the report the objective
A grader that checks section completion and arithmetic can accept a recommendation whose premises are unsupported. That outcome identifies a gap in the checks. Establishing whether an agent exploited the reward or followed an unsuitable learned routine requires evidence from its available context, actions and repeated attempts.
If later decisions use an unsupported assumption, the error can affect subsequent calculations and actions. These are failure modes we want to test; their empirical status is recorded on the limitations page.
An early grading failure
Our earlier grader exposed a related failure in what we rewarded. According to the person who ran the test, it gave 87 of 87 to a submission with correct arithmetic, an unsupported funding curve, a false hedge explanation and blanket approvals for everything. The genuine submission scored 85. The original receipt is not published; the grading history keeps this reported failure separate from the later reproducible fixture tests.
That report concerns a grader accepting bad work. We use it as evidence of a grading failure; the learning mechanism requires a separate test. It motivated checks on the sources supporting each conclusion, the permissions for each action and the obligations remaining after execution.
What the role requires
A quote can satisfy the client's price limit and still fail its deadline or permitted legal form. The trader must establish which dependency prevents execution and decide whether to change the structure, escalate or decline. The case connects the following responsibilities through documents, constraints and dated records.
| Function | Work represented in the case |
|---|---|
| Client and treasury | Identify the required amount, currency and deadline, permitted structures and available funding sources. |
| Credit and markets | Calculate value and the counterparty, curve, collateral and maturity exposures remaining after a proposed hedge. |
| Capital and liquidity | Check capital and available cash separately; trace a payment to funds in the relevant account. |
| Legal and compliance | Check rights, mandate limits and effective approvals; distinguish requests, responses and settlement. |
| Accounting and control | Reconcile carry, valuation changes, reserve movements and fees under the case’s accounting treatment. |
| Operations | Check cash and title transfers against deadlines, reconcile receipts and retain unresolved claims. |
These are authored dependencies in the synthetic task. The environment does not simulate negotiations with separate human departments.
Why tasks like this are scarce
We think public evaluations contain relatively few tasks that follow financial obligations across time; we have not measured their prevalence. This task records client delivery, collateral rights and payment capacity across twenty days of changing facts. Reviewing that behaviour requires the sequence of actions and book states, including the information available at each decision.
A snapshot case supports analysis at a specified point. The daily environment adds observations of how accepted actions affect later decisions.
The book
The environment is one synthetic funding book. The fictional Meridian Bank owns an issued AAA-rated note, called U, and has a separate USD1bn legacy borrowing. It sells U to a funding provider for USD1bn, receives U's total return through a twenty-year total return swap, and pays the swap's financing leg. It then lends the USD1bn to a different borrower for five years.
| Part of the book | What it does |
|---|---|
| Sale of U | The provider gets legal title. Meridian gets USD1bn in cash. |
| Twenty-year return swap | Meridian receives U's coupons and price changes, including negative ones, and pays the contracted financing rate. The swap advances no second principal. |
| Five-year loan | Meridian lends the proceeds to an emerging-market borrower and receives interest and, if it performs, principal at year five. |
| Legacy borrowing | The original USD1bn debt stays outstanding under its own agreement. The new transaction does not repay it. |
Cash, title and return
At inception
- Funding providerBank
USD 1,000,000,000.00 in cash
The provider buys the reference note. The sale supplies the cash.
- BankFunding provider
1,000,000,000.00 units of U
Legal ownership transfers to the provider. This dashed arrow is title, not another cash payment.
- BankLoan borrower
USD 1,000,000,000.00 loan advance
The bank lends the sale proceeds to the borrower. The swap supplies no second principal payment.
During the funding contract
- Funding providerBank
U's total return
The bank receives positive return and owes negative return under the total return swap. It keeps the note's economic risk.
- BankFunding provider
OIS + 70 basis points
The bank pays this annual financing rate. OIS is the overnight indexed swap benchmark. One basis point is 0.01 percentage points.
Why the bank still records the note
The provider owns U, but the bank retains its return. Under this fixture's accounting treatment, the bank still records USD 1,000,000,000.00 of the note and a matching sale-financing liability of USD 1,000,000,000.00. Accounting recognition is not legal ownership and does not permit a second sale.
The diagram separates the swap's return and financing legs for explanation. The fixture settles them net each quarter; these arrows are not extra gross cash receipts.
The synthetic quotes make the twenty-year financing cheaper than a matching replacement: 70 basis points locked against a 100 basis point replacement quote, worth about USD41m at inception. This is a modelled valuation under the case's assumptions. Holding other inputs fixed, wider replacement spreads increase it and tighter spreads reduce it. The swap also passes through U's price changes. Valuation changes and cash settlements are recorded separately, and unpaid contractual amounts remain outstanding. The funding option goes through this position in detail.
The maturities do not match. When the loan repays at year five, fifteen years of funding obligations remain. The base case parks the returned principal at the overnight rate; putting it back to work needs a new quote, a credit decision and an executable contract. If the desk buys the five-year protection offered against the provider at inception, that hedge also expires fifteen years before the funding does.
The standalone base illustration is also short of capital: statutory equity after the setup fee is USD99m against a whole-book requirement of about USD135.65m. Finding that USD36.65m shortfall is part of that case. Generated runtime families disclose their own opening capital and are checked separately; each capital check uses the equity and requirement for its own scenario.
The institutions, jurisdictions, instruments, opinions, prices and event paths are fictional. The official sources cited in the case support particular rules; they do not authenticate the case or approve a real transaction.
What the agent does each day
The registered engineering paths observe twenty daily checkpoints; an actual model attempt has its own frozen observation and resource limits. At a checkpoint the agent receives the current documents, quote board, state of the book and its hash. It submits a typed batch of actions and calculations against that hash. Invalid actions leave the book unchanged. Valid execution writes dated receipts; a fatal payment cutoff commits only the executed prefix and stops later activity. The next packet inherits those positions, unpaid obligations and promises.
The action set is deliberately concrete. The agent can hold capital, execute a collateral agreement, substitute collateral, execute an offset trade and consolidate netting, buy protection on the provider, allocate inventory to repo, draw or repay a facility, meet a margin call, settle a payable, and escalate or decline with a scoped rule and typed premises. Every action is priced by the public quote board and every accepted action leaves a receipt. The agent cannot invent a funding curve, sell a credit index or buy an arbitrary asset, because those are not on the board.
Approval requests and responses have separate dated records. Execution depends on the effective response and the applicable legal and financial conditions. Declining an unworkable route can be a supported decision even when the client remains unfunded.
The next step depends on those facts. A cheap funding quote with a pending approval calls for a different decision from the same quote with effective approval and cash ready to settle. If approval arrives after the client's deadline, the original route may no longer solve the problem. The agent must then assess the available alternatives. The checks can accept different decisions when their premises meet the stated support requirements.
What the score measures, and what it does not
How the book is scored explains the correctness and client-outcome checks, the separate qualification verdict and the limits of treating those results as management performance.
Where to go next
Read What the first grading checks missed, then The funding option, How the book is scored, Designing against reward hacking and How the environment is designed. Reference: what the grader recomputes keeps the equations together. Try a decision on the Practice desk.