On October 24, 2026, a new evaluation benchmark called TauIndianBankBench was officially released on the arXiv platform (paper ID 2610.30161v1). This benchmark focuses on the Indian retail banking domain and aims to more accurately assess the compliance performance of tool‑invoking conversational agents in real‑world business workflows.
TauIndianBankBench includes 1000 simulated customer assistance tasks, covering typical operations such as reporting a lost bank card or closing a deposit account. Unlike conventional benchmarks that only check the system’s record at the end of the conversation, it simultaneously verifies whether the agent has performed all prescribed steps (e.g., “a fixed-term deposit must first obtain a final settlement quote before being closed”) and confirms that no changes not requested by the user have been introduced.
Research indicates that, if judged solely on the final record, 6.5% to 12.0% of sessions would be incorrectly classified as successful—despite actual issues such as skipping required steps or arbitrarily modifying states. This finding reveals that current evaluation methods lack sensitivity to procedural compliance, potentially misleading assessments of a model’s reliability when deployed in financial scenarios.