Engine-level metrics like speculative decoding acceptance length and per-replica token latency quantiles are provided
The metrics you actually need to debug inference issues — such as speculative decoding acceptance length and per-replica, engine-side token latency quantiles — are automatically provided in a dashboard.
Éléments favorables
Introducing Modal Auto Endpoints: Optimized inference you actually own | Modal Blog
Extrait original
We don't hide the metrics . The metrics you actually need to debug inference issues, like speculative decoding acceptance length and per-replica, engine-side token latency quantiles, are automatically provided in a dashboard. Low bar, but we didn't put it there!