Skip to content

Bonus Assignment: Take an assistant into operations

This optional, ungraded assignment reviews Domain 4 by putting one small AI application through all three task statements: watching it, costing it, and locking it down. It is hands-on, so the steps below are written to be run in your own AWS account or a sandbox account, because operating something you can see is what makes this review stick. Budget about thirty minutes for the hands-on path.

If no account is available, every step has a paper equivalent noted at the end, and the reasoning is the part that matters either way.

Disclaimer

This project is for educational purposes only and might lead to AWS service charges. You assume any risk associated with this project. You are responsible for any costs associated with your AWS account.

The scenario

AnyCompany Support launched an assistant that answers agent questions from a library of resolution notes, and it runs beside an older recommendation model on a hosted endpoint. Two weeks in, nobody can say whether the assistant is still answering well, the bill has grown for reasons no one has named, and the security review is scheduled for Friday. You are the engineer who has to have answers for all three.

Steps

Task 4.1: Monitor ML and AI model inference and performance

  • Open the monitoring views for your model invocations and write down the three measures you would put on a wall for this assistant, then say what each one would look like on a bad day.
  • Design the drift detection pipeline for it on one page: what the baseline is, what is captured from live traffic, how the two are compared, and what threshold triggers an alert. Name who is paged.
  • Say which of your measures would move if a tool the assistant calls started failing, and which would not move at all.
  • Sketch the A/B comparison you would run to replace the recommendation model, including how much traffic the candidate gets and which metric decides the outcome.

Task 4.2: Optimize and manage ML and AI infrastructure costs and performance

  • In a cost report, group spend by service and then by tag. Note which of your resources cannot be attributed to a team yet, and tag one of them.
  • Identify what the assistant's invocations are billed for, then list the levers available on that bill and pick the one you would try first. Write the quality risk it carries.
  • For the recommendation endpoint, read its utilization over a representative window and decide whether it has a sizing problem or a scaling problem. Justify which.
  • Name the purchasing option that fits the endpoint's steady baseline and say precisely which usage that option covers. Then name a commitment that does not cover it, and why.
  • Set a budget with an alert for this workload, and choose the amount by reasoning about the workload rather than by picking a round number.

Task 4.3: Secure ML and AI workloads and model endpoints

  • Read the role your application uses to reach the model, and remove one permission it does not need. Note how you confirmed it was unnecessary.
  • Decide which credential type the assistant should use in production, and write one sentence on why the other types were rejected.
  • Draw the network path from the application to the model API and mark the point where traffic would leave the private network. If it does leave, say what would keep it inside.
  • Configure a safeguard that masks personal information in responses, then send a prompt designed to produce a masked value and confirm the behavior you expected.
  • Check whether your activity record would show a model invocation call. If it would not, say what has to be configured, and decide whether this workload needs it.

Check yourself

Monitoring coverage

Could you say which signal would have caught each of the three failures in the scenario, and which failure your current monitoring would miss entirely?

Two bills, two levers

Could you explain to a colleague why the lever that fixed the endpoint bill does nothing to the assistant bill, using what each one is charged for?

Coverage, precisely

Could you name which purchasing option covers machine learning instance usage and which covers general compute, without hedging on either?

Content against access

Could you state the difference between a control that decides who may call the model and a control that decides what the response may contain, and why one does not substitute for the other?

Find your gap

If any step felt unclear, return to the skills in that task's section before moving on. Note the gap so you know what to review.

If you have no AWS account

Every step has a paper equivalent. Describe the dashboard and the drift pipeline instead of building them, reason about utilization from a described traffic pattern instead of reading a chart, and write the safeguard policy and the role change out instead of applying them. The decisions are the review; the console is only where they become concrete.

Clean up

Delete anything you created, so nothing is left running or stored: remove any endpoint, any safeguard configuration, any budget, and any tags or resources added for this exercise. If you used a generative AI tool to produce sample notes, clear its conversation history so the sample data is not retained.