AI 客栈
返回首页

AWS 用两个实测量替换 TTS 扩缩容的猜测阈值

  • 问题定义：TTS 请求单次占用连接数秒，没有 token 生成循环可挂指标，默认按吞吐或并发均值设阈值会静默失败——保守则 GPU 空转多付钱，激进则排队越过 SLO，两种都不触发告警。
  • 测量方法：在 ml.g5.xlarge 上部署 Kokoro-82M 自定义容器，先以闭环负载阶梯测单实例并发上限 Q_max（3 秒首字节 SLO 下为 50，第 60 档越界），再强制触发一次扩容测全程耗时 T_total（实测 260 秒，加上策略检测上限 60 秒为 320 秒）。
  • 推导结果：由 surge_ratio=1.25 算出扩容阈值 37.5、缩容阈值 25；扩容冷却 30 秒、缩容冷却 960 秒（约为 T_total 的三倍），容器自身以 queue_max_depth=50 对过载请求直接返回 503。
  • 已知取舍：实例供给占 T_total 的 56%，缩小镜像或用热池比改应用代码更有效；从 2 台缩到 1 台会把幸存实例顶到 Q_max，长冷却只能压制反复扩缩而非结构性消除。
  • 行业意义：这套「SLO + surge ratio + Q_max + T_total」四数模型不限于 TTS，可迁移到任何延迟敏感、并发受限的实时端点；关键在于扩缩容目标值来自对自家端点的测量，而非从别的负载形态照搬。 🔗 AWS Machine Learning Blog
Diagram showing a client connecting to an Amazon SageMaker AI real-time endpoint, which contains a custom GPU container exposing two routes, POST /invocations for response streaming and WS /invocations-bidirectional-stream for bidirectional streaming, both feeding into a KPipeline running Kokoro-82MDiagram showing three AWS CDK stacks in sequence, SpeechFoundationStack, SpeechModelCacheStack, and SpeechEndpointStack, feeding into a decision gate labeled scaling_enabled AND a feasible measured plan artifact exists, branching to EndpointAutoscaling plus EndpointObservability on yes, or no scalinDiagram showing five tools in sequence, tts-bench drift labeled read-only seconds, tts-bench qmax labeled mutates about 36 minutes, tts-bench ttotal labeled mutates up to about 28 minutes, tts-bench plan labeled read-only seconds, and tts-bench validate labeled read-only seconds, connected by edgesLine chart showing p95 time to first audio byte in milliseconds rising with concurrency from 65.4 ms at concurrency 1 to 3374.8 ms at concurrency 60, crossing a dashed SLO line at 3000 milliseconds between concurrency 50 and 60, with Q_max equals 50 annotated at the last point that still meets the SStacked horizontal bar chart showing eight stages of a scale-out timeline for Kokoro-82M on ml.g5.xlarge, a 30 second capacity request stage, a 181 second dominant provisioning stage, several small stages for container start and warmup, a 21 second in-service stage, and a 47 second traffic recoveryJoseph Cottingham