Skip to content

Bonus Assignment: Choose and defend a modeling approach

This optional, ungraded assignment reviews Domain 2 by taking two requests from the same fictitious company through all three task statements:

  • the approach decision
  • the training or customization that follows from it
  • the evidence that the result works

Disclaimer

This project is for educational purposes only and might lead to AWS service charges. You assume any risk associated with this project. You are responsible for any costs associated with your AWS account.

The scenario

AnyCompany Retail brings you two requests in the same week. Merchandising wants weekly demand forecasts per store, and it has years of its own sales history to work from. Customer service wants an assistant that answers agent questions from a returns policy that legal revises frequently. Nobody has told you which approach either request needs, and that is the assignment.

Steps

Task 2.1: Choose modeling approaches

  • Write one sentence per request naming the approach you would take: a rule, a managed AI service, a trained model, or a foundation model, and say what in the request forced that choice.
  • For the forecast, name the requirement that constrains the algorithm choice more than accuracy does, and say which algorithm family you would rule out because of it.
  • For the assistant, decide between retrieval and customization, then list the parts a production retrieval system needs and mark which ones a managed retrieval service would hand you.
  • Name one requirement you would have to give up in each request, choosing from output quality, latency, and cost, and say who in the business absorbs that loss.

Task 2.2: Train, fine-tune, and customize models

  • For the forecast, decide whether the training code should come from a built-in algorithm, a script of your own on a prebuilt framework container, or a container you build, and justify it in one sentence.
  • Pick three hyperparameters you would put into a tuning job, declare a range for each, and name the single objective metric the job optimizes. State whether that metric is minimized or maximized.
  • Write the prompt for the assistant with explicit instructions, a stated output format, and two worked examples. Then run it, and note exactly what still falls short.
  • Decide from that result whether customization is warranted. If it is, name the technique and the artifact it would require you to prepare. If it is not, say what made prompting sufficient.

Task 2.3: Analyze and evaluate performance

  • Record the forecast experiment as a run with its parameters, metrics, and artifacts, and write the baseline figure you would compare a later run against.
  • Name one metric that would show the forecast model drifting, and say whether it measures the incoming data, the prediction quality, or the endpoint's behavior.
  • Evaluate three assistant answers two ways, once against a reference answer and once for whether the retrieved passage actually contained the fact. Note any answer that reads well and still failed.
  • Describe how you would compare a revised assistant prompt against the current one without changing what any user receives, and name what you would log to make that comparison honest.

Check yourself

Approach reasoning

Could you defend each approach to somebody who proposed using one foundation model for both requests, using the requirements rather than preference?

Prompt before customization

Could you state what specifically failed in your prompt, and why that failure does or does not warrant a customization job?

Metric direction

Could you say whether your objective metric is minimized or maximized, and what a tuning job would optimize toward if you pointed it the wrong way?

Retrieval versus generation

Could you say which of your assistant's weak answers was a retrieval failure and which was a generation failure, and how you told them apart?

Find your gap

If any step felt unclear, return to the skills in that task's section before moving on. Note the gap so you know what to review.

If you have no AWS account

Every step has a paper equivalent. Describe the training route and the tuning ranges instead of submitting a job, write the prompt and predict where it would fall short instead of running it, and reason about which metric would move instead of measuring it. The decisions are the review; an account is only where they become concrete.

Clean up

Delete anything you created, so nothing is left running or stored: remove any training job output and model artifacts, delete any endpoint or variant you deployed, remove any evaluation job output and the objects it wrote, and delete any prompt or retrieval index you created for the assistant. If you used a generative AI tool to produce sample data, clear its conversation history so the sample data is not retained.