Skip to content

Task Statement 1.3: Describe the AI/ML Development Lifecycle

AWS AIF-C01 Exam Focus: Task 1.3 tests your end-to-end understanding of how AI/ML solutions progress from business problem definition to production monitoring. You must know the stages of an AI/ML pipeline, foundation model sourcing strategies, deployment options, AWS service mappings across stages, core MLOps principles, and evaluation metrics.

1. End-to-End AI/ML Development Pipeline

┌──────────────────────────┐    ┌──────────────────────────┐    ┌──────────────────────────┐
│ 1. Problem Formulation   │───>│ 2. Data Preparation &    │───>│ 3. Model Selection &     │
│    & Requirements        │    │    Feature Engineering   │    │    Training              │
└──────────────────────────┘    └──────────────────────────┘    └──────────────────────────┘
             ▲                                                               │
             │                                                               ▼
┌──────────────────────────┐    ┌──────────────────────────┐    ┌──────────────────────────┐
│ 6. Production MLOps &    │<───│ 5. Model Deployment &    │<───│ 4. Model Evaluation &    │
│    Drift Monitoring      │    │    Serving               │    │    Hyperparameter Tuning │
└──────────────────────────┘    └──────────────────────────┘    └──────────────────────────┘

The AI/ML development lifecycle is an iterative process composed of six distinct phases:

Pipeline Stage Key Activities Output Deliverables
1. Problem Formulation Define business objectives, SLA constraints, and ROI goals; select the appropriate AI paradigm. Business metric baseline, project scope document.
2. Data Prep & Engineering Clean raw data, remove PII, construct features, and generate vector embeddings. Cleaned datasets, feature store tables, vector stores.
3. Selection & Training Choose model architecture, run distributed training, or prepare fine-tuning data. Trained model weights, prompt templates.
4. Evaluation & Tuning Evaluate holdout set performance, tune hyperparameters, perform human review (RLHF). Validation performance reports, optimal hyperparameter set.
5. Deployment & Serving Host model behind production endpoints (Real-Time, Serverless, Async, or Batch). Production API endpoints, inference microservices.
6. MLOps & Monitoring Audit data drift, latency, and operational cost; automate retraining loops. CloudWatch alerts, automated CI/CD retraining pipelines.

2. Sources of Foundation Models (FMs)

FM Source Type Operational Model AWS Implementation Best For
Managed Pre-Trained FMs Closed-source models served entirely via serverless API calls. Zero infrastructure setup. Amazon Bedrock (Anthropic Claude, Meta Llama, Amazon Nova) Rapid deployment, lowest operational effort, pay-per-token model.
Open-Weight Pre-Trained FMs Public model weights hosted on self-managed compute infrastructure. Amazon SageMaker JumpStart, Hugging Face on AWS Scenarios requiring deep inspection of model weights or strict offline VPC hosting.
Custom Fine-Tuned FMs Pre-trained FMs adapted on domain-specific datasets (e.g., medical, legal). Amazon Bedrock Custom Models, SageMaker JumpStart Fine-Tuning Adapting model tone, output format, or domain accuracy without building a model from scratch.
FMs Trained from Scratch Full pre-training of a new model starting from randomly initialized weights. Amazon SageMaker AI on AWS Trainium / EC2 UltraClusters Organizations with massive data and capital building proprietary foundation models (highest cost/effort).

3. Production Model Deployment Strategies

1. Managed API Service (Serverless / Managed Endpoints)

  • How It Works: AWS manages the underlying GPU infrastructure, scaling, patching, and availability. You consume the model via REST/gRPC APIs.
  • AWS Services: Amazon Bedrock, Amazon Comprehend, Amazon Q.
  • Pros: Zero infrastructure overhead, instant scalability, fastest time-to-market.
  • Cons: Less control over underlying hardware configuration, vendor-imposed quota limits.

2. Self-Hosted / Managed Infrastructure Endpoints

  • How It Works: You select specific compute instances (GPUs/Inferentia) and deploy model containers onto infrastructure managed via a platform service.
  • AWS Services: Amazon SageMaker Endpoints (Real-Time, Async, Serverless), Amazon ECS / EKS.
  • Pros: Direct control over instance types, hardware accelerators, concurrency limits, and custom container runtimes.
  • Cons: Requires active capacity planning, autoscaling configuration, and infrastructure monitoring.

4. AWS Service Mapping Across the Lifecycle

Pipeline Stage Primary AWS Services Service Function
Data Preparation & Labeling Amazon S3, AWS Glue, SageMaker Data Wrangler, Amazon Ground Truth Scalable object storage, automated ETL, feature engineering, and human-in-the-loop data labeling.
FM Sourcing & App Building Amazon Bedrock, SageMaker JumpStart, Amazon Q, Kiro Discovering pre-trained FMs, managed RAG implementation, and AI-assisted development.
Custom Training & Tuning Amazon SageMaker AI, AWS Trainium, AWS EC2 UltraClusters Distributed custom training, hyperparameter optimization, and custom hardware acceleration.
Deployment & Hosting Amazon Bedrock Endpoints, SageMaker Real-Time / Serverless / Async Endpoints Hosting serverless APIs or dedicated infrastructure endpoints for inference.
MLOps & Pipeline Automation SageMaker Pipelines, SageMaker Model Registry, AWS CodePipeline Orchestrating automated ML workflows, tracking model version lineage, and CI/CD deployment.
Monitoring & Governance SageMaker Model Monitor, SageMaker Clarify, Amazon CloudWatch, Bedrock Model Evaluation Auditing data drift, model bias, hardware latency, and benchmark evaluation.

5. Core MLOps Concepts & Production Readiness

  • Experimentation & Reproducibility: Tracking code versions, datasets, hyperparameters, and model output artifacts using SageMaker Experiments to ensure all training runs can be audited and reproduced.
  • Model Registry: A centralized repository (SageMaker Model Registry) for versioning models, tracking data lineage, and managing formal approval workflows (e.g., Staging $\rightarrow$ Production).
  • Data Drift vs. Concept Drift:
  • Data Drift: Statistical properties of incoming input features ($X$) change over time (e.g., demographic shifts in incoming user data).
  • Concept Drift: Statistical relationship between input features ($X$) and target outputs ($Y$) changes over time (e.g., consumer purchasing patterns shifting permanently post-economic event).
  • Model Monitoring & Retraining:
  • Using SageMaker Model Monitor to continuously audit live inference requests for data quality degradation and drift.
  • Triggering automated retraining pipelines (SageMaker Pipelines) when drift metrics breach configured threshold limits.
  • Managing Technical Debt: Eliminating unversioned datasets, ad-hoc manual scripts, and unmonitored endpoints through Infrastructure-as-Code (IaC) and automated CI/CD pipelines.

6. Model Evaluation Metrics: ML vs. Business

Statistical Machine Learning Metrics

Classification Confusion Matrix Fundamentals

  • True Positive (TP): Correctly predicted positive class.
  • False Positive (FP): Type I Error (Predicted positive, actual was negative).
  • False Negative (FN): Type II Error (Predicted negative, actual was positive).
  • True Negative (TN): Correctly predicted negative class.

Classification Formulae & Decision Rules

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

  • Accuracy: Proportion of total correct predictions. Use ONLY on balanced datasets.

$$\text{Precision} = \frac{TP}{TP + FP}$$

  • Precision: Measures how many predicted positives were actually positive. Use when False Positives carry high cost (e.g., Spam Filter flagging critical business emails).

$$\text{Recall (Sensitivity)} = \frac{TP}{TP + FN}$$

  • Recall: Measures how many actual positive cases the model caught. Use when False Negatives are catastrophic (e.g., Fraud Detection, Medical Screening).

$$F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

  • F1 Score: Harmonic mean of Precision and Recall. Use when evaluating imbalanced datasets.

Text Generation & Natural Language Metrics

  • ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Evaluates n-gram overlap between generated text and reference text. Primary metric for Text Summarization**.
  • BLEU (Bilingual Evaluation Understudy): Evaluates n-gram precision of generated text against reference translations. Primary metric for Machine Translation**.

Business Value Metrics

  • Return on Investment (ROI): Net financial gain generated by the AI solution relative to implementation and compute hosting costs.
  • Cost Per Interaction / Token Cost: Total API/compute spend divided by total processed user requests.
  • Customer Lifetime Value (CLV) & Conversion Rate: Business impact of recommendation engines on user acquisition and sales completion.
  • Customer Satisfaction Score (CSAT / NPS): Explicit user feedback (e.g., thumbs up/down, survey scores) evaluating AI application outputs.

7. Exam Decision Rules & Common Pitfalls

High-Yield Decision Rules

  1. Select Recall when failing to detect a positive case causes severe risk or financial loss (e.g., fraud detection, disease identification).
  2. Select Precision when false alarms disrupt user workflows or damage trust (e.g., blocking legitimate customer accounts, spam classification).
  3. Select F1 Score when evaluating models on imbalanced datasets requiring a balance between Precision and Recall.
  4. Select Managed API (Amazon Bedrock) over self-hosted endpoints when the goal is zero infrastructure management, rapid prototyping, and token-based pricing.
  5. Select SageMaker Model Monitor when you need to automatically identify input data drift or accuracy loss on production endpoints.
  6. Select SageMaker Pipelines to automate the sequence of data processing, training, evaluation, and registration steps in CI/CD workflows.
  7. Select ROUGE to evaluate summarization models; select BLEU to evaluate translation models.

Common Exam Traps

  • Trap 1: Choosing Accuracy for Imbalanced Data
  • If 99% of transactions are legitimate and 1% are fraudulent, a dummy model predicting "never fraud" yields 99% accuracy but fails completely. Always choose Precision, Recall, or F1 Score for imbalanced data.
  • Trap 2: Confusing Data Drift with Concept Drift
  • Data Drift = Input feature distributions shift over time.
  • Concept Drift = The mathematical relationship between input features and target outcomes changes.
  • Trap 3: Selecting Custom Pre-Training over Fine-Tuning
  • Training a foundation model from scratch requires millions of dollars and massive data compute. If a scenario asks to adapt an FM to custom terminology with limited budget, select Fine-Tuning or RAG.