返回首页

AWS 公布 Bedrock AgentCore 九阶段 Agent 评估框架:准确率 93%→96%,回归发现压至 4 小时内

  • 框架结构:横跨 tool / agent / 多智能体三个层级,覆盖 safety、quality、performance、cost、operational control 五个维度;九个阶段从「定义质量」到「自动改进」构成单一闭环,全部跑在 AgentCore Evaluations 上,配合 Bedrock Guardrails、CloudWatch、S3/Redshift、Lambda 自定义评估器。
  • 量化效果:三个月定向迭代把准确率从 93% 提升至 96%,纯幻觉从每万条 138 次降至 52 次以下,未授权计算从 59 降至 44 以下;生产环境质量回归的发现时间由数天至数周压缩至 4 小时内,首次信号是带 AgentCore trace 的 CloudWatch 告警。
  • 关键机制:CI 用分钟级 sentinel 集拦截回归——一次数据规范化更新导致实体解析失败,部署被自动阻断;CD gate 在冻结的工具行为上对 200 道留出题做端到端打分,平均分改善但单题通过率下降同样不放行;生产侧以 30 分钟为一个评估窗口、连续三次越线才告警,一次数据同步不匹配窗口曾使通过率从 78% 掉到 38%。
  • 行业意义:AWS 将其称为 AgentOps——评估不是最终 QA 步骤,而是 Agent 自身的运行时操作系统。对多子智能体系统而言,「每个工具都有自己的评估契约、每层各有黄金集」意味着质量控制点从单模型输出前移到工具、编排与合成三个可独立失效的面。

🔗 AWS Machine Learning Blog

Figure 1 Workflow diagram showing 9-stage AgentCore evaluation journey from Define to ImproveFigure 3. Side-by-side comparison of two candidate agent responses in an AgentCore On-Demand Evaluation, scored across quality, latency, and cost. The numbers and values in the figure are illustrative.Figure 4. Evaluation scorecard showing per-category failure breakdown and pass rate trend. The numbers and values in the figure are illustrative.Figure 5. AgentCore execution trace for the Customer Insights Agent. The left panel shows the full span tree across tool calls (illustrated through example tool names) and the invoke_agent Strands Agents step, with per-step latencies. The trajectory diagram shows the reasoning graph from invocationDashboard screenshot showing agent evaluation metrics with 97.3% pass rate across 183 questions. Displays KPIs for latency, grounding rate, and error types. Includes custom evaluators tracking hallucinations, wrong customer/period errors, and supportError reporting dashboard showing 183 evaluated questions with 97.3% pass rate. Bar chart displays error categories: Metric Swap 8 instances, Symbol Hallucination 5, Unauthorized Calc 5, and others. Includes week-over-week comparison with green