system-type-ml-serving Skill

Source

• Creator: microsoft
• Repository: microsoft/amplifier-bundle-systems-design
• Skill file: skills/system-type-ml-serving/SKILL.md

What it does

Domain patterns for ML/AI serving and training systems — model serving, feature stores, training pipelines, experiment tracking, A/B testing, GPU scheduling, and failure modes. Use when designing or evaluating machine learning infrastructure, model serving platforms, or AI-powered product features.

Skill instructions

System Type: ML/AI Serving & Training Patterns, failure modes, and anti-patterns for machine learning infrastructure and model serving systems. --- Model Serving Patterns Online Inference (Real-Time) What it is. Model receives a request, runs inference synchronously, returns a prediction. The model sits in the request path — latency matters as much as accuracy. When to use. User-facing predictions where the result must be immediate: search ranking, recommendation, fraud scoring at transaction time, autocomplete. When to avoid. When predictions can be precomputed. When the model is too large to meet latency budgets. When the cost per prediction doesn't justify real-time serving. Key concerns. Tail latency (P99, not just P50) dominates user experience. Model loading time creates cold start problems. Memory footprint determines how many models fit per node. Timeouts must be set aggressively — a slow prediction is worse than a fallback. Batch Inference What it is. Run predictions over a la