Skip to content

Skill 2.3.1: Perform Reproducible Experiments

Task 2.3: ML Model and Foundation Model (FM) Development – Analyze and evaluate the performance of ML and AI systems

Core services: Amazon SageMaker AI with MLflow · Amazon Bedrock Evaluations · Amazon Bedrock Prompt Management

Core Objective & Mental Model

  • Business objective: Make every ML/GenAI experiment reproducible, auditable, and comparable so decisions rest on evidence, not memory.
  • Reproducibility triad: Version code, data, environment; log parameters, metrics, artifacts; record seeds and dependencies.
  • GenAI twist: Prompts and retrieved context are first-class inputs; version them like code.
  • Exam lens: If a scenario asks “why two runs differ” or “compare variants”, choose tracking/versioning/evaluation—not more compute.

Service Selection Matrix

Need Primary Service Why Distractor Trap
Track traditional ML training runs SageMaker AI + MLflow Managed MLflow logs params/metrics/artifacts, model registry, lineage CloudWatch only has metrics/logs, not params/artifacts
Evaluate GenAI models/prompts/RAG Amazon Bedrock Evaluations Automatic, LLM-as-judge, human; built-in/custom datasets; RAG metrics Prompt Management stores prompts but does not score
Version and compare prompts Amazon Bedrock Prompt Management Variables, versions, variants; tracked prompt changes Editing prompt in place destroys reproducibility
Trace GenAI app steps MLflow tracing on SageMaker AI Captures inputs/outputs/metadata per step CloudWatch logs are operational, not experiment trace
Monitor live baseline/drift CloudWatch + SageMaker Model Monitor Compare against known-good baseline over time Offline evaluation is not drift detection

SageMaker AI with MLflow – Reproducible Tracking

  • Managed MLflow: tracking server, experiments, runs, params, metrics, artifacts, tags, source, model registry.
  • Artifact store: S3; encryption: KMS; auth: IAM; private access: VPC endpoints.
  • Autologging: captures framework params/metrics/models; still log data version, seeds, and dependencies manually.
  • Model Registry: version, stage, approve models; link to runs for lineage.
  • Tracing: for GenAI, records each step (retrieval, prompt, LLM call) with inputs/outputs/metadata.
  • Reproducibility checklist: code commit · data snapshot/version · hyperparameters · random seeds · container/environment · dependencies · metrics · artifacts.

Exam Gotcha: MLflow tracking server requires an S3 artifact store and an IAM role. Without artifact logging, runs are not reproducible. Exam Gotcha: MLflow on SageMaker AI is fully managed—not self-managed MLflow on EC2. It handles tracking server, IAM, and S3 integration.

Amazon Bedrock Evaluations – Reproducible GenAI Assessment

  • Three modes:
  • Automatic/programmatic: computes metrics over a dataset; no human UI.
  • LLM-as-a-judge: judge model scores and explains responses.
  • Human: workers rate prompts/responses; console-created human jobs require a browser-facing display.
  • Evaluation targets: models · prompt variants · Knowledge Bases (RAG).
  • Metrics:
  • Text generation: BLEU, ROUGE, BERTScore, semantic similarity.
  • RAG: faithfulness, answer relevance, context relevance, context recall, context precision.
  • Safety: toxicity, bias, robustness.
  • Datasets: built-in or custom JSONL in S3; results to S3; metrics to CloudWatch.
  • Reproducibility: same dataset + same prompt version + same model config + same judge = comparable scores.

Exam Gotcha: To isolate RAG failures, evaluate retrieval (context relevance/recall) separately from generation (faithfulness/answer relevance). Exam Gotcha: Bedrock evaluation jobs require model access enabled and an IAM role with S3 and Bedrock permissions.

Amazon Bedrock Prompt Management – Versioned Prompt Experiments

  • Prompt as artifact: store prompt text, variables, model config, versions.
  • Variables: parameterize prompts for reuse across datasets/experiments.
  • Versions: immutable snapshots; roll back and compare.
  • Variants: compare wording changes; integrate with Bedrock Evaluations.
  • Reproducibility: pin prompt version in evaluation; never edit in place.

Exam Gotcha: Prompt Management does not execute live traffic or score outputs. It stores/versions prompts; pair with Bedrock Evaluations for scoring.

Baselines, Drift & Shadow (Adjacent Reproducibility)

  • Baseline: record known-good metrics over a period; use CloudWatch for operational metrics, Model Monitor for data/quality drift.
  • Drift: departure from baseline in input distribution, prediction quality, or endpoint metrics.
  • Shadow variant: SageMaker AI shadow test routes a copy of live traffic to candidate; production answers caller; candidate responses logged for offline comparison.
  • A/B vs shadow: A/B sends real traffic to candidate; shadow does not affect caller.

Exam Gotcha: Shadow testing validates live request patterns without user impact; A/B testing exposes users to candidate responses. Exam Gotcha: Some endpoint configurations are excluded from shadow testing—do not assume it is always available.

Security & Compliance

  • IAM roles: least privilege for MLflow tracking server, Bedrock evaluation jobs, S3 access.
  • Encryption: KMS at rest for S3 artifacts, MLflow tracking server, Bedrock evaluation results; TLS in transit.
  • Network: VPC endpoints for SageMaker, S3, Bedrock where private access is required.
  • Data privacy: human evaluation may expose data to workers; use private workforce or anonymize.
  • Audit: CloudTrail for API activity; CloudWatch Logs for job logs; MLflow run lineage for model governance.

Anti-Patterns & Exam Traps

  • No tracking: relying on memory/notebooks; cannot reproduce or compare.
  • Cherry-picking: reporting only best run; hides variance.
  • Editing prompts in place: loses version history; breaks reproducibility.
  • Evaluating on training data: inflated metrics; no holdout.
  • Single metric for GenAI: ignores semantic correctness; use overlap + embedding + judge/human.
  • Black-box RAG evaluation: only final answer; cannot isolate retrieval vs generation.
  • Assuming shadow testing always available: some endpoint configurations excluded.
  • Using CloudWatch alone for experiment tracking: metrics/logs are not params/artifacts/lineage.
  • Ignoring seeds/dependencies: same code can produce different results.

Quick Decision Matrix

Scenario Best Fit Why
Compare two training runs and identify differences MLflow on SageMaker AI Logs params/metrics/artifacts/lineage
Score GenAI outputs against reference dataset Bedrock Evaluations (automatic) Programmatic metrics at scale
Need expert judgment on sensitive outputs Bedrock Evaluations (human) Human-in-the-loop ratings
Compare prompt wording versions Prompt Management + Evaluations Versioned prompts + scored variants
Validate candidate on live traffic without user impact SageMaker AI shadow test Copy traffic, log only
Monitor production drift CloudWatch + Model Monitor Baseline comparison over time
Trace a GenAI app step-by-step MLflow tracing Inputs/outputs per step

Key Metric Cheat Sheet

Metric Type Examples Use When
Classification Accuracy, precision, recall, F1, AUC Traditional ML
Regression RMSE, MAE, R² Traditional ML
Text overlap BLEU, ROUGE Reference-based generation
Embedding BERTScore, semantic similarity Meaning-based generation
RAG Faithfulness, answer relevance, context relevance/recall/precision Retrieval + generation
Safety Toxicity, bias, robustness Responsible AI
Operational Latency, error rate, throughput Shadow/production