Skip to content

Skill 1.1.1: Extract data from data sources

Question 1 - Traditional ML Evaluation Metrics

A financial services company has developed a fraud detection model using Amazon SageMaker to classify credit card transactions as fraudulent or legitimate. The dataset is highly imbalanced, with fraudulent transactions representing only 0.3% of the total records. The compliance team requires that the model be evaluated with a metric that focuses on the model's ability to correctly identify fraudulent transactions (the positive class), while accounting for both false positives and false negatives.

Which ONE metric should the data science team use as the primary evaluation metric?

A) Accuracy
B) Precision
C) Recall
D) F1 Score
E) Root Mean Squared Error (RMSE)

Click to reveal Answer & Explanation

Correct Answer: D - F1 Score

Correct Options Analysis

  • D is Correct: The F1 Score is the harmonic mean of Precision and Recall. For imbalanced classification problems like fraud detection (0.3% positive class), accuracy is misleading because a model that always predicts "non-fraud" would achieve 99.7% accuracy while detecting zero fraud. F1 Score balances the trade-off between false positives (precision) and false negatives (recall), making it the most defensible single metric.

Distractor Analysis

  • A is Incorrect: Accuracy is inappropriate for severely imbalanced datasets. A majority-class classifier would report >99% accuracy but fail entirely at fraud detection.
  • B is Incorrect: Precision measures the proportion of true positives among all positive predictions. While it penalizes false positives, it ignores false negatives (missed fraud), which carry high financial risk.
  • C is Incorrect: Recall measures the proportion of actual frauds detected. Optimizing recall alone could lead to a model that flags nearly everything as fraud, causing excessive customer friction and operational costs.
  • E is Incorrect: RMSE is a regression metric for continuous outputs, not applicable to a binary classification task.

Key Takeaway: For imbalanced classification problems, always pair Precision with Recall (via F1 Score or PR-AUC) rather than relying on Accuracy alone.

Question 2 - LLM-as-a-Judge Evaluation

A healthcare technology startup is building a medical symptom triage assistant using Amazon Bedrock with Claude models. The assistant must generate responses that are medically safe, contextually appropriate, and empathetic. The team wants to automate quality evaluation at scale without hiring expensive medical reviewers for every output. They need a solution that can assess subjective qualities like tone, helpfulness, and harmlessness.

Which TWO approaches would best meet this requirement with minimal human effort? (Select TWO)

A) Use Amazon Bedrock Evaluations with an LLM-as-a-judge model to score responses against predefined criteria like helpfulness, harmlessness, and tone quality.
B) Implement programmatic evaluation only, using BLEU and ROUGE scores against a reference answer set.
C) Set up a periodic HumanInTheLoop review where all outputs are rated by a single subject matter expert.
D) Use Amazon Bedrock Evaluations with multiple judge criteria including instruction adherence, medical safety, and context relevance.
E) Rely solely on exact-match string comparison between generated responses and a golden answer key.

Click to reveal Answer & Explanation

Correct Answers: A and D

Correct Options Analysis

  • A is Correct: Amazon Bedrock Evaluations supports LLM-as-a-judge, where a judge model (e.g., Claude) evaluates generation quality against custom rubric criteria. This automates subjective evaluation metrics like tone and helpfulness.
  • D is Correct: Multiple judge criteria with independent scoring (helpfulness, harmlessness, correctness) provides a comprehensive automated evaluation framework, reducing human reviewer dependency.

Distractor Analysis

  • B is Incorrect: BLEU and ROUGE measure n-gram overlap against reference text. They fail to capture semantic correctness, empathy, or safety for open-ended conversational outputs.
  • C is Incorrect: Human review by a single SME does not scale, introduces bias from one evaluator, and fails the "minimal human effort" constraint.
  • E is Incorrect: Exact-match comparison penalizes valid paraphrases and different wordings, which is inappropriate for creative generative output.

Key Takeaway: Use Amazon Bedrock Evaluations with LLM-as-a-judge for subjective quality attributes at scale; reserve human evaluation only for audits or edge cases.

Question 3 - RAG Retrieval Accuracy Assessment

A legal document search platform has deployed a RAG application using Amazon Bedrock Knowledge Bases and a vector store in Amazon OpenSearch Serverless. Users report that answers are fluent but sometimes missing key clauses from relevant contracts. The team must isolate whether the problem is in retrieval quality or generation quality.

Which TWO evaluation queries would help isolate the root cause? (Select TWO)

A) Compare retrieved context chunks against ground-truth relevant passages using Context Precision and Context Recall.
B) Measure Faithfulness of the final answer against the retrieved context only.
C) Check overall answer length and token counts.
D) Evaluate Answer Relevance against the user query.
E) Measure end-to-end system latency at peak traffic.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: Context Precision and Context Recall directly quantify whether the retriever returned the right passages. If recall is low, missing clauses are a retrieval failure.
  • B is Correct: Faithfulness measures whether the generated answer factually supports the context actually retrieved. If retrieval works (high Context Recall) but Faithfulness is low, the generator is dropping facts, isolating generation as the culprit.

Distractor Analysis

  • C is Incorrect: Token count reflects output length, not factual completeness or retrieval accuracy.
  • D is Incorrect: Answer Relevance measures how well the answer addresses the query, but does not compare against retrieved context, so it cannot distinguish missing retrieval from generation issues.
  • E is Incorrect: Latency is an operational metric unrelated to retrieval accuracy or answer completeness.

Key Takeaway: Evaluate RAG systems in layers—measure retrieval metrics (Context Precision/Recall) independently from generation metrics (Faithfulness/AnswerRelevance) to pinpoint the failure stage.

Question 4 - Shadow Variant Deployment

A large e-commerce platform is testing a new product recommendation model trained on Amazon SageMaker. The team wants to validate the new model's latency and prediction quality under real production traffic conditions before serving it to customers. The current production endpoint serves millions of requests per hour, and no customer-facing change is permitted during testing.

Which ONE approach meets these requirements?

A) Deploy the new model as a separate endpoint behind a load balancer routing 50% of traffic to it.
B) Deploy a shadow variant on the production endpoint that asynchronously receives a copy of live inference requests, while production responses continue to be returned to callers.
C) Withhold a subset of the training data to score the candidate model offline.
D) Replace the production variant with the new model on Saturday night when traffic is lowest.

Click to reveal Answer & Explanation

Correct Answer: B

Correct Options Analysis

  • B is Correct: Amazon SageMaker Shadow Testing deploys the candidate model as a shadow variant. Production traffic is mirrored to the shadow, but only production responses are sent to callers. This enables latency, error rate, and prediction quality comparison under realistic load with zero impact to customer experience.

Distractor Analysis

  • A is Incorrect: Routing 50% of live traffic (A/B testing) exposes users to the new model's outputs, violating the "no customer-facing change" constraint.
  • C is Incorrect: Offline scoring on held-out data cannot capture real-time request patterns, payload sizes, concurrency levels, or network latencies encountered in production.
  • D is Incorrect: Replacing the model during low-traffic windows still changes what customers callers receive, and low-traffic periods may not be representative of the model's behavior under peak load.

Key Takeaway: SageMaker Shadow Tests enable live-traffic validation without affecting caller responses—ideal for pre-deployment reality checks on latency, errors, and prediction behavior.

Question 5 - Reproducible Experiment Tracking with MLflow

A data science team at a manufacturing company uses SageMaker AI to train quality-inspection models for factory defect detection. Two engineers on the team report different validation accuracies for what they claim is the same model configuration. Management requires a system that tracks parameters, metrics, artifacts, and code versions for every training run to enable reproducibility.

Which TWO AWS capabilities should the team configure? (Select TWO)

A) Enable SageMaker MLflow tracking to log every experiment run with its hyperparameters, metrics, and model artifacts.
B) Store model artifacts in Amazon S3 with a timestamp in the filename and no other metadata.
C) Register models in the SageMaker Model Registry after tracking runs.
D) Print training log output to CloudWatch without structured parameter logging.
E) Perform the training locally on each engineer's machine and paste results into a shared spreadsheet.

Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: SageMaker MLflow provides fully managed experiment tracking—recording parameters (e.g., learning rate, epochs), metrics (accuracy, loss), artifacts (trained weights), and commit/environment info per run. This makes run-to-run differences visible.
  • C is Correct: SageMaker Model Registry stores model artifacts with versioning, metadata, and approval status. This combined with MLflow tracking ensures a complete lineage from experiment to deployable artifact.

Distractor Analysis

  • B is Incorrect: Timestamped object names in S3 alone do not track hyperparameters, code dependencies, or dataset versions, making it impossible to reproduce the run.
  • D is Incorrect: CloudWatch logs without structured parameter tracking cannot be diffed across runs; changes in hyperparameters or data are not associated with the metric outputs.
  • E is Incorrect: Local training spreadsheets cannot capture environment differences (Python library versions, random seeds), which are common causes of non-reproducible results.

Key Takeaway: Pair MLflow with SageMaker Model Registry to achieve full lineage: tracked parameters/metrics during experiments and versioned artifacts at deployment time.

Question 6 - NLP Evaluation Metrics (BLEU, ROUGE, BERTScore)

A multilingual customer support team has built a translation and summarization pipeline in Amazon Bedrock that generates summaries of support tickets in multiple languages. The team must evaluate generated summaries for semantic similarity against human-written reference summaries. They note that different languages may express the same meaning with very different vocabulary.

Which TWO metrics are most appropriate for this cross-lingual semantic evaluation? (Select TWO)

A) BLEU
B) ROUGE-1
C) BERTScore
D) Semantic similarity using embeddings
E) Exact Match

Click to reveal Answer & Explanation

Correct Answers: C and D

Correct Options Analysis

  • C is Correct: BERTScore computes token-level similarity using contextual embeddings, allowing it to assess semantic equivalence across different phrasings and languages, beyond simple n-gram overlap.
  • D is Correct: Semantic similarity using embeddings (e.g., cosine similarity between sentence embeddings) measures meaning rather than surface-form word overlap, making it well-suited for cross-lingual or paraphrased outputs.

Distractor Analysis

  • A is Incorrect: BLEU is based on n-gram precision against a single reference; it performs poorly in cross-lingual settings because exact phrase matches across languages are unlikely, even when meaning is equivalent.
  • B is Incorrect: ROUGE-1 counts unigram overlap with reference summaries; it fails when languages or synonyms vary heavily, underestimating semantically correct but lexically different outputs.
  • E is Incorrect: Exact match essentially never occurs in free-form text generation and is unsuitable for summarization or translation evaluation.

Key Takeaway: For semantic evaluation of generated text—especially cross-lingual or paraphrased—use embedding-based metrics (BERTScore, semantic similarity) rather than surface-level overlap scores (BLEU, ROUGE).

Question 7 - Human Evaluation Frameworks

A legal tech company has launched a contract-review RAG assistant using Amazon Bedrock. The output of the assistant provides legal interpretations that are sent directly to client-facing attorneys. The risk management team requires that every response with a low confidence score be reviewed by a human attorney, and that responses rated poorly be fed back into the evaluation pipeline as training data.

Which TWO architectural decisions satisfy these requirements? (Select TWO)

A) Build an Amazon Bedrock human evaluation workflow where a human workforce (e.g., partner attorneys) reviews sampled outputs and rates them.
B) Use Amazon Bedrock Evaluations with human workers, creating a work team and submitting tasks to them for rating when model confidence is below a threshold.
C) Automate flagging using only an LLM-as-a-judge with no human oversight step.
D) Log all poor ratings into a feedback dataset stored in S3 for later evaluation and model fine-tuning.
E) Delete all low-confidence responses automatically and return a generic error to the client.

Click to reveal Answer & Explanation

Correct Answers: B and D

Correct Options Analysis

  • B is Correct: Amazon Bedrock Evaluations supports human-based evaluation jobs where human workers are assigned tasks with prompts and responses. This satisfies the requirement that low-confidence responses be routed to human reviewers.
  • D is Correct: Storing human ratings as structured data (e.g., JSON with response text and rating) allows the team to analyze failure patterns, compute evaluation metrics, and fine-tune future model versions.

Distractor Analysis

  • A is Incorrect: While human review is conceptually correct, "build a workflow" is vague and does not specify the managed AWS service. Option B properly uses Bedrock Evaluations with human workers.
  • C is Incorrect: Automating flagging with only an LLM-as-a-judge skips the required human review step for low-confidence responses, violating the risk management constraint.
  • E is Incorrect: Silently deleting low-confidence outputs deprives users of service and fails to collect feedback data for improvement.

Key Takeaway: Use Amazon Bedrock Evaluations with human workers for high-risk use cases requiring judgment, and store human ratings as structured data for iterative improvement.

Question 8 - Drift Detection and Baseline Monitoring

A retail company has deployed a demand-forecasting model in SageMaker to predict daily sales for each of its 500 stores. The model performed well at launch, but the ML team suspects that recent inventory shortages and holiday-season effects have changed the underlying data patterns. They need to detect when the model's inputs diverge from the training data distribution and alert on quality degradation.

Which TWO SageMaker capabilities address this requirement? (Select TWO)

A) Enable SageMaker Model Monitor with a data-quality baseline to compare live inference inputs against the training distribution and alert on deviations.
B) Enable SageMaker Model Monitor with a model-quality baseline to compare model predictions against ground-truth labels when they become available.
C) Enable CloudWatch Container Insights only to monitor CPU utilization of the endpoint instances.
D) Disable statistical tests and rely on manual review of a 5% random sample of inference logs.
E) Retrain the model only once per year during the annual planning cycle.

Click to reveal Answer & Explanation

Correct Answers: A and B

Correct Options Analysis

  • A is Correct: SageMaker Model Monitor's data-quality baseline compares input feature distributions (e.g., mean, std) to the baseline captured at training time. Alarms can trigger when drift statistics exceed thresholds, detecting covariate drift from inventory shortages or seasonal inputs.
  • B is Correct: Model-quality monitoring evaluates prediction accuracy against actual ground truth (e.g., actual sales after the forecast period). This captures concept drift where input distributions look similar but the underlying relationships have shifted.

Distractor Analysis

  • C is Incorrect: CPU utilization metrics reveal infrastructure load, not input distribution shifts or prediction quality degradation.
  • D is Incorrect: Manual sampling of 5% of logs is statistically weak, labor-intensive, and does not scale to real-time detection; it also violates the alert-on-degradation requirement.
  • E is Incorrect: Annual retraining ignores gradual degradation between training cycles. Drift detection is needed precisely to identify when the model should be retrained sooner.

Key Takeaway: Use SageMaker Model Monitor with BOTH data-quality (input drift) and model-quality (output drift) baselines for comprehensive drift detection with alerts.

Question 9 - Debugging Training Convergence

A deep learning team is training an image classification model on SageMaker using 1,000,000 images. After several epochs, they observe that the training loss is decreasing while the validation loss is increasing. The training accuracy is 98%, but validation accuracy plateaus at 74%.

Which TWO techniques are most likely to address this overfitting problem? (Select TWO)

A) Increase L2 regularization strength on the model's weights.
B) Increase the learning rate so the optimizer converges faster.
C) Add dropout layers between fully connected layers to reduce co-adaptation of neurons.
D) Increase the number of hidden layers to increase model capacity.
E) Stop logging validation metrics and report only training accuracy.

Click to reveal Answer & Explanation

Correct Answers: A and C

Correct Options Analysis

  • A is Correct: Increasing L2 (ridge) regularization penalizes large weights and prevents the model from fitting noise or memorizing specific training examples, reducing the gap between training and validation performance.
  • C is Correct: Dropout randomly disables neurons during training, forcing the network to learn redundant and more generalizable representations. This is a standard remedy for overfitting in deep networks.

Distractor Analysis

  • B is Incorrect: A higher learning rate generally worsens overfitting by allowing the optimizer to jump to sharper minima that fit training data more tightly.
  • D is Incorrect: Adding more layers increases model capacity, making overfitting worse by giving the network more parameters to memorize training examples.
  • E is Incorrect: Ignoring validation metrics does not solve the underlying overfitting problem; it hides it and leads to a model that fails in production.

Key Takeaway: When training loss decreases but validation loss increases, the model is overfitting—apply regularization (L2, dropout) or early stopping rather than increasing capacity or learning rate.

Question 10 - Content Quality and Bias Detection

A public-sector agency has deployed a document-generation system using Amazon Bedrock that produces policy summaries for the public. A recent audit found that certain demographic groups consistently received summaries with less detailed information than other groups, likely due to discret bias in the training data. The team must proactively detect bias and ensure content quality across all demographic categories before publication.

Which THREE capabilities should the team configure? (Select THREE)

A) Use Amazon Bedrock Guardrails to filter outputs for harmful or biased content based on custom deny policies.
B) Configure Amazon Bedrock Evaluations with a judge model to rate responses for demographic parity and content quality consistency.
C) Use only programmatic metrics (BLEU, ROUGE) to assess whether summaries have equal length across demographic groups.
D) Store evaluation outputs in a data lake to perform periodic bias audits using statistical tests for group parity.
E) Use a human evaluation workflow in Amazon Bedrock with a diverse reviewer pool to manually verify that summaries are equable across demographic categories.

Click to reveal Answer & Explanation

Correct Answers: A, B, and D

Correct Options Analysis

  • A is Correct: Amazon Bedrock Guardrails provides configurable safety filters (e.g., deny lists, sensitive-information filters, and custom policies) that can block or flag biased content patterns at inference time.
  • B is Correct: An LLM-as-a-judge in Bedrock Evaluations can be prompted to explicitly evaluate parity and quality consistency across demographic groups, providing scalable automated bias detection.
  • D is Correct: Persisting evaluation outputs in a data lake (e.g., S3) enables periodic offline statistical audits, enabling the ML team to quantify bias (e.g., group mean differences) and track improvements over time.

Distractor Analysis

  • C is Incorrect: BLEU and ROUGE are surface-level overlap metrics. Equal-length summaries do not guarantee unbiased or high-quality content for each demographic group.
  • E is Incorrect: While human evaluation is useful, relying solely on human reviewers is costly, slow, and vulnerable to reviewer bias. A combination of automated and human evaluation is more robust.

Key Takeaway: For bias detection in GenAI outputs, combine Guardrails (real-time filtering), an LLM-as-a-judge (automated evaluation), and periodic audits with stored outputs for statistical verification.