Abstract
Tool integration enables language models to affect external systems, while introducing uncertainty involving permissions, retries, timing, and outcomes. This article formulates agent processes as state transitions with explicit preconditions and postconditions, discussing why a timeout cannot simply be treated as a failure followed by a retry, and why task success must be checked against external state. We add a conditional reliability analysis and protocols for idempotency and outcome reconciliation, and propose an evaluation matrix covering rejection, delays, lost acknowledgments, and concurrent modifications. Task benchmark results reported in the papers belong to their original authors; this site has conducted no new live experiments with autonomous agents.[1] ReAct: Synergizing Reasoning and Acting in Language Mod[2] Toolformer: Language Models Can Teach Themselves to Use[3] WebArena: A Realistic Web Environment for Building Auto
English abstract
Tool-using agents are analyzed as distributed decision systems. The review distinguishes proposed actions, admitted operations and externally confirmed outcomes; derives a simple chain-reliability model; and specifies how idempotency and reconciliation handle uncertain responses. Task success must be checked against external state rather than an agent's own narration. The workflow diagrams are engineering analyses, not evidence of autonomous deployment.
1. ReAct introduces an updatable observation loop
ReAct organizes reasoning and actions in alternation, allowing external observations to influence the next step. Compared with generating a complete plan in one pass, this structure provides a place to check environmental changes and adjust actions. It does not automatically resolve missing observations, misread pages, or incorrect premises; if the system does not check the actual results returned by tools, the loop may merely become a continuous narration of failure.[1] ReAct: Synergizing Reasoning and Acting in Language Mod
2. Toolformer learns when and how to make calls
Toolformer studies the selection of useful API calls for training material through self-supervision, enabling the model to learn tool-use behavior. It addresses learning the ability to call tools, not the complete permissions regime of a production environment. Generating correct API parameters, obtaining account authorization, confirming side effects, and handling retries are conditions at different levels. Deployed systems must still control these boundaries outside the model.[2] Toolformer: Language Models Can Teach Themselves to Use
3. WebArena tests tasks against target states
WebArena provides runnable website environments and long-horizon tasks, focusing evaluation on whether functional outcomes satisfy user intent. Two operational paths may reach the same valid terminal state even if their wording differs; conversely, action descriptions may resemble examples while no data have actually been saved. The paper's historical baselines are specific to its models, environments, and point in time; this article does not use those results to infer the capabilities of current browser agents.[3] WebArena: A Realistic Web Environment for Building Auto
4. Why long tasks need recovery mechanisms
Assume that each step has a constant success probability p conditional on all preceding steps having succeeded, and that failure at any step causes the entire task to fail. The probability that all n steps succeed is then p to the power of n. Figure 2 in this version uses p=0.90, 0.97, and 0.99 to illustrate the product of a conditional chain. Retries, correlated faults, recovery actions, and branches in real systems change these conditional probabilities, so the curves cannot directly estimate actual success rates; they indicate that complete tasks should be measured, not just individual steps.作者分析与复核建议 / Author analysis
5. How observations and external content should enter decision-making
This article recommends storing user goals, system permissions, and external material separately. Webpages, documents, and tool outputs are objects of observation; even if they say “ignore the original requirements,” this must not change task authorization. For account modifications, message sending, or publishing, the execution layer should decide based on existing permissions and an explicit task scope, rather than treating page text as new instructions from the user. This is a system design recommendation.作者分析与复核建议 / Author analysis
6. Retries must account for side effects
Retrying after a failed read differs from retrying after a write that may already have succeeded. Submission actions require an idempotency identifier or a verifiable outcome state; if a timeout leaves the outcome unknown, the actual state should be queried first. Inferring “the operation did not occur” solely from “the call did not return success” can produce duplicate content or transactions. Tool interfaces should distinguish failure, success, and unknown outcomes.作者分析与复核建议 / Author analysis
7. What evidence acceptance checks should examine
A task record should specify at least the initial state, target state, permitted actions, stopping conditions, and acceptance method. Success cannot rely solely on the agent's own account; it should be confirmed through an independent read, a readback after saving, or an executable check within the environment. Timeouts, recovery counts, and human interventions should also be reported. If a human must correct a critical step, the result should be recorded as completion through human–agent collaboration, not fully automatic success.[3] WebArena: A Realistic Web Environment for Building Auto
State confirmation and linguistic confirmation must not be conflated
In website tasks, a page displaying “saving” and data having been persisted are different states. Reliable acceptance checks should be able to observe the target value through a fresh read after saving, checking a version or transaction identifier when necessary. For irreversible actions, confirmation before execution and readback afterward serve different purposes and cannot replace each other. Several consecutive actions that appear reasonable still cannot prove task completion without evidence of the final state.作者分析与复核建议 / Author analysis
8. Actions, admission, and outcomes require distinct states
A model outputting a tool call indicates only a proposed intent; a permissions service accepting the request indicates that it passed admission checks at that time; only a final external state matching the goal constitutes evidence of completion. A state machine should distinguish at least not sent, sent but unconfirmed, confirmed success, confirmed rejection, and requiring human handling. Each transition is bound to an acting principal, a parameter digest, and a target version, preventing subsequent steps from treating an acknowledgment for another object as the result of the current operation. Natural-language statements of “completed” should not directly drive business state.[1] ReAct: Synergizing Reasoning and Acting in Language Mod[2] Toolformer: Language Models Can Teach Themselves to Use
9. Timeouts produce unknown outcomes
When the server has committed a transaction but its response is lost, the timeout seen by the client may be identical to a timeout occurring before the server has executed the operation. Blindly generating a new request may duplicate side effects. An idempotency key must be bound to the same intent and parameters, with the result stored by the server; if parameters change under the same key, the request should be rejected rather than overwrite the existing record. When state is unknown, first query the target version or operation acknowledgment, then decide whether to explicitly replay with the same key. These are distributed systems design principles, which cannot be replaced by “do not repeat” in a model prompt.[1] ReAct: Synergizing Reasoning and Acting in Language Mod[3] WebArena: A Realistic Web Environment for Building Auto
10. How weaknesses accumulate in multistep tasks
If all stages have conditional success probability p and are independent, the probability that all L steps succeed is p^L. Actual tool workflows are usually not independent: early state errors change subsequent inputs, while shared login or network infrastructure introduces common failures. Thus, this equation is only an analytical reference; a more general expression should use a sequence of conditional probabilities. Evaluation should observe final outcomes and preserve failure categories for each stage, distinguishing planning errors, parameter errors, permission denials, and external service anomalies.[3] WebArena: A Realistic Web Environment for Building Auto[2] Toolformer: Language Models Can Teach Themselves to Use
11. Outcome evaluation requires external state and budgets
A web agent may click the correct button without actually submitting, or may change other fields that should have remained unchanged. Acceptance checks should verify the task-specified state and protected state rather than rely solely on a success notification in the interface. Tool-call counts, model resource consumption, latency, and human intervention must also be recorded. System-level decoding acceleration can shorten one stage, but does not fix state-machine or business-validation errors; comparisons should attribute the contributions of model decisions, tool protocols, and runtime systems separately.[3] WebArena: A Realistic Web Environment for Building Auto[4] Fast Inference from Transformers via Speculative Decodi
Core equations and derivations
(1) The general chain decomposition does not require independence. It can be simplified to p to the power of L only when each step's success probability is fixed and the steps are independent.
LaTeX
P(S_1,\ldots,S_L)=\prod_{i=1}^{L}P(S_i\mid S_1,\ldots,S_{i-1})(2) An idempotency identifier should be associated with an explicit execution principal and an intent digest; this is a protocol constraint, not proof through a content hash that an action has actually been completed.
LaTeX
\mathrm{IntentID}\longmapsto H(\mathrm{actor},\mathrm{target},\mathrm{version},\mathrm{parameters})(3) Task evaluation considers both intended changes and states that must not change. Checking rules for both should be defined before execution.
LaTeX
\mathrm{Success}=\mathrm{TargetSatisfied}\ \land\ \mathrm{ProtectedStatePreserved}Method and evidence workflow
Analytical plots and numerical examples
Recomputable worked example
If each step has success probability 0.97 conditional on prior success, the success rate of an entire 20-step chain is approximately 0.544. Recovery, retries, and external state can change these conditional probabilities, so this number cannot be used to estimate the performance of actual agent products.
Comparison of methods
| Method or judgment | Controlled variables and measurements | Failure conditions and explanations |
|---|---|---|
| Model self-report of completion | External system state, target version, and operation acknowledgment | Self-report does not constitute evidence of actual execution |
| Retry after a timeout | Retain the original intent identifier; query first, then replay with the same key | Resending with a new key may duplicate side effects |
| Long-chain success rate | Classification of stagewise conditional failures and common faults | p^L cannot replace the actual correlation structure |
| Task benchmark | Verify both target and protected states, accounting for budgets | A success button or a few examples cannot establish general reliability |
Verification and experimental plan
- Predefine task goals, states that must not change, permitted tools, and minimum permissions; do not treat external webpage content as system instructions.
- Establish an intent identifier, parameter digest, and target version for every write action; explicitly reject different parameters under the same key.
- Inject failures before commit, lost acknowledgments after commit, and delayed responses; verify that unknown outcomes remain pending reconciliation rather than being automatically declared successes or failures.
- Test version conflicts when two executors modify the target concurrently, preventing an older action whose response arrives later from overwriting a newer state.
- Independently read external results at task completion and check protected state; retain records of rejection, human intervention, and partial completion.
- When aggregating success rates, also report tool-call counts, model resource consumption, latency, and stratification by task difficulty, avoiding the presentation of only selected successful trajectories.
Scope and limitations
- Tool outcomes depend on external systems and permission conditions.
- The independent-stage curve is only an analytical reference.
- Idempotency requires server-side protocol support.
- No new results on real-world external adoption of agents or autonomous deployment are presented.
Mapping primary sources to claims
- ReAct: Synergizing Reasoning and Acting in Language ModelsYao et al. · 2022-10-06v3; method and experiments
- Toolformer: Language Models Can Teach Themselves to Use ToolsSchick et al. · 2023-02-09abstract; training tool-use behavior
- WebArena: A Realistic Web Environment for Building Autonomous AgentsZhou et al. · 2023-07-25v4; abstract, benchmark and functional evaluation
- Fast Inference from Transformers via Speculative Decoding, , · 2022-11-30Abstract; 原文方法、理论和实验章节见作者原始论文链接
Citation and manuscript export
BibTeXRISCSL JSONTXTArticle JSON
Author, version, and sources
AI-assisted research review; Codex (AI) wrote the article and performed analytical derivations based on the listed primary literature. This site has no record of independent peer review or original model experiments for this article.
Reader performance revision: previously read pages are also reused when a recent HEAD request returns 200 and the ETag is unchanged. The main text, equations, sources, and scientific-figure data retain the content of the previous version.
The conceptual cover is separate from the scientific diagrams; scientific figures are calculated from public equations and explicit assumptions, and all results from original papers are attributed.
Author, version, and correction history · /guides/agents-tools-and-outcomes/v5/
Sources for the cover and scientific figures · Data, sources, and assumptions · Publication and version history
Source revision: 6251a250b27bb8b449f59b392f81abd355383770