Domain 4: Operations (GenAIOps)
Evaluation (AutoSxS)
GenAIOps
1 / 3
❓
What is AutoSxS?
(Click to reveal)💡
Automatic Side-by-Side evaluation
Objective "judge" model compares outputs of two models
Helps determine if Model A is better than Model B.
Objective "judge" model compares outputs of two models
Helps determine if Model A is better than Model B.
- Problem: How do you know if Model A is better than Model B?
- Solution: Vertex AI AutoSxS (Automatic Side-by-Side). An objective "judge" model compares the outputs of two models based on your criteria.
Deployment & Monitoring
- Endpoints: Models must be deployed to an Endpoint to be accessible via API
- Safety: Monitoring for "Model Drift" or "Hallucination rates" over time
Quick Reference: Limits & Quotas
| Resource | Default Logic |
|---|---|
| Max Output Tokens | Varies by model (e.g., 2048 or 8192) |
| Context Window | Gemini 1.5 Pro supports up to 1M+ tokens |
| Data Privacy | Google does not train foundation models on customer data |