Skill 1.1.3: Troubleshoot and debug data ingestion and storage issues that involve capacity and scalability

Question	Answer & Explanation
### Question 1 - Kinesis Data Streams Consumer Bottleneck An organization is ingesting real-time sensor data into an Amazon Kinesis Data Stream configured in Provisioned capacity mode with 50 shards. The application that processes the data and writes it to an S3 data lake is falling behind, and the `GetRecords.IteratorAgeMilliseconds` metric is steadily climbing. The team notices the consumer application is frequently hitting the `ReadProvisionedThroughputExceeded` exception. The solution must minimize operational overhead and avoid modifying the producer applications. Which action should the ML engineer take to resolve this issue? - A) Increase the number of Kinesis shards. - B) Switch the Kinesis Data Stream to On-Demand capacity mode. - C) Enable enhanced fan-out for the Kinesis consumers. - D) Migrate the ingestion pipeline to Amazon Managed Streaming for Apache Kafka (Amazon MSK).	Correct Answer: C Correct Answer Analysis: - C is Correct: `ReadProvisionedThroughputExceeded` indicates the consumer is exceeding the standard 2 MB/s read limit per shard. Enabling enhanced fan-out provides each consumer with a dedicated 2 MB/s read throughput per shard (up to 5 consumers), eliminating read contention without requiring stream re-sharding or producer code changes. Distractor Analysis: - A is Incorrect: Increasing shards (re-sharding) increases the overall stream capacity but requires updating the consumer logic to handle the new shard distribution, and does not inherently solve read contention if multiple consumers are competing for the standard 2 MB/s limit. - B is Incorrect: Switching to On-Demand mode automatically scales the stream to accommodate varying write throughput, but it does not change the standard 2 MB/s read limit per shard for standard consumers. - C is Incorrect: Migrating to MSK introduces massive operational overhead and architectural changes, which violates the least operational overhead constraint. > Exam Tip / AWS Best Practice: For Kinesis Data Streams, if consumers are throttled on read throughput, enable Enhanced Fan-Out before attempting to add more shards.
### Question 2 - Streaming Pipeline Downstream Write Bottleneck A machine learning team is using Amazon Managed Service for Apache Flink to process streaming data from an Amazon Kinesis Data Stream and load it into an Amazon OpenSearch Service cluster. The `GetRecords.IteratorAgeMilliseconds` metric on the Kinesis stream is steadily increasing. The team observes that the Flink application CPU utilization is low, but the OpenSearch cluster is consistently reporting `ClusterStatusRed` or high thread pool queue sizes. The pipeline must be optimized with the least infrastructure management overhead. What is the root cause of the ingestion lag, and how should it be resolved? - A) The Flink application is under-provisioned; increase the KPU (Kinesis Processing Units) allocated to the Flink application. - B) The Kinesis stream is shard-throttled; switch the stream to On-Demand capacity mode. - C) The OpenSearch cluster write capacity is the bottleneck; scale the OpenSearch cluster by adding nodes or increasing instance size. - D) The Kinesis consumers are hitting read limits; enable Kinesis Enhanced Fan-Out.	Correct Answer: C Correct Answer Analysis: - C is Correct: The low CPU utilization of the Flink application combined with high thread pool queues on OpenSearch indicates that the processing job is waiting on the downstream write target. Adding consumer compute capacity (KPUs) will not help because the bottleneck is at the storage layer (OpenSearch). Distractor Analysis: - A is Incorrect: Adding KPUs to the Flink app will not fix the issue because the Flink app is not CPU-bound; it is blocked waiting for OpenSearch to acknowledge writes. - B is Incorrect: The Kinesis stream is successfully holding the data (evidenced by the growing IteratorAge, not WriteProvisionedThroughputExceeded). The issue is the consumer reading it out, not the producers writing to it. - D is Incorrect: Enhanced Fan-Out solves read throttling on the Kinesis side, but the Flink application is not being throttled by Kinesis; it is blocked downstream. > Exam Tip / AWS Best Practice: If consumer CPU is low but IteratorAge is climbing, investigate the downstream write target. Scaling consumers only moves the queue; it does not increase overall pipeline throughput if the target is the bottleneck.
### Question 3 - S3 Small File Problem in Batch Ingestion A data engineering team runs a daily AWS Glue Spark job to ingest raw user activity logs from Amazon S3, transform them, and prepare them for ML model training. The raw data consists of millions of small JSON files (average size 1 KB). Lately, the Glue job is taking an excessively long time to complete, and the execution profile shows heavy task scheduling overhead with very little data read per task. The team must resolve this performance issue while minimizing compute costs. Which approach will resolve this capacity and scalability issue? - A) Increase the number of DPUs (Data Processing Units) allocated to the Glue ETL job. - B) Use Amazon EMR instead of AWS Glue to handle the large volume of files. - C) Configure the Glue job to use the `groupFiles` and `groupSize` parameters to combine small files into larger splits. - D) Convert the JSON files to Parquet format after the Glue job completes.	Correct Answer: C Correct Answer Analysis: - C is Correct: The "small file problem" in Spark causes massive overhead because Spark creates a task for each file. Using AWS Glue's `groupFiles` (e.g., `groupFiles: "inPartition"`) and `groupSize` parameters combines small files into larger groups for processing, drastically reducing task scheduling overhead and improving performance without requiring more compute. Distractor Analysis: - A is Incorrect: Increasing DPUs adds more compute, but it does not solve the architectural inefficiency of reading millions of tiny files. It would incur higher costs without fundamentally fixing the task overhead bottleneck. - B is Incorrect: Moving to Amazon EMR introduces significant administrative overhead and does not inherently solve the Spark small files problem without tuning. - D is Incorrect: Converting to Parquet is a good practice for downstream reads, but the ingestion job itself is failing due to reading the JSON files. The small files must be grouped before or during the read phase. > Exam Tip / AWS Best Practice: When processing millions of small files in AWS Glue, use `groupFiles` and `groupSize` to minimize Spark task overhead and avoid throwing compute at an I/O inefficiency.
### Question 4 - OpenSearch Serverless Capacity for Vector Ingestion An AI team is using Amazon OpenSearch Serverless to store vector embeddings for a massive Retrieval-Augmented Generation (RAG) workload. They are ingesting 50 million embeddings via a high-throughput batch job. During ingestion, the team encounters frequent `WriteThrottlingException` errors, and the ingestion pipeline fails. The team must complete the ingestion as quickly as possible with minimal operational tuning. What is causing the throttling, and how can it be resolved? - A) The OpenSearch Serverless collection has reached its storage capacity limit; delete old indices to free up space. - B) The OpenSearch Serverless collection's OCU (OpenSearch Compute Unit) scaling limit is too low; increase the maximum OCU count in the capacity policy. - C) The client is sending synchronous requests; switch to asynchronous requests using the OpenSearch Bulk API. - D) The vector embeddings exceed the maximum dimension size allowed by OpenSearch Serverless; reduce the embedding dimensions.	Correct Answer: B Correct Answer Analysis: - B is Correct: OpenSearch Serverless automatically scales based on OCU (OpenSearch Compute Unit) usage. However, if the maximum OCUs are capped at the default minimum in the capacity policy, heavy batch ingestion will cause `WriteThrottlingException`. Increasing the maximum OCU limit allows the collection to scale out to handle the high write throughput. Distractor Analysis: - A is Incorrect: Storage capacity limits do not trigger `WriteThrottlingException`; they trigger out-of-space errors. - C is Incorrect: While the Bulk API is recommended for performance, the throttling is a server-side compute limit (OCU). Even with the Bulk API, if OCUs are capped, the requests will throttle. - D is Incorrect: OpenSearch Serverless supports large dimensions (up to 16,000). Invalid dimensions would cause a validation error at index creation, not a write throttle. > Exam Tip / AWS Best Practice: OpenSearch Serverless scales via OCUs. If you face `WriteThrottlingException` during heavy batch ingestion, check and increase the maximum OCU limit in the capacity policy.
### Question 5 - SageMaker Feature Store Ingestion Throttling A retail company is ingesting real-time user session features into an Amazon SageMaker Feature Store Online store. During peak traffic hours, the ingestion pipeline, running on AWS Lambda, fails with `ThrottlingException` errors from the SageMaker Feature Store API. The business requires extremely low latency for online feature ingestion and cannot afford data loss. What should the ML engineer do to resolve this issue with the least operational overhead? - A) Increase the concurrency limit of the AWS Lambda functions running the ingestion pipeline. - B) Implement an exponential backoff and retry mechanism in the Lambda function. - C) Convert the real-time ingestion pipeline to a batch process using AWS Glue. - D) Contact AWS Support to request a quota increase for the SageMaker Feature Store Online store PUT requests.	Correct Answer: D Correct Answer Analysis: - D is Correct: The `ThrottlingException` specifically indicates that the rate of requests to the SageMaker Feature Store Online store has exceeded the account's default TPS (Transactions Per Second) quota. Requesting a quota increase directly addresses the capacity bottleneck at the service level. Distractor Analysis: - A is Incorrect: Increasing Lambda concurrency will only generate more requests against the already-throttled Feature Store endpoint, worsening the throttling and potentially causing cascading failures. - B is Incorrect: Exponential backoff helps handle transient throttling gracefully, but it does not solve the capacity issue during peak hours and introduces latency to the feature ingestion, potentially missing the real-time requirement. - C is Incorrect: Converting to a batch process violates the real-time ingestion requirement for the online store. > Exam Tip / AWS Best Practice: Service-level `ThrottlingException` errors during peak ingestion usually indicate an account TPS quota limit. Apply for a quota increase rather than just tuning the client compute.
### Question 6 - Distributed Training Shared File System Performance A deep learning training job is running on a 10-node Amazon SageMaker Distributed Training cluster. The training data consists of millions of high-resolution images stored in an Amazon S3 bucket. The training script downloads the images sequentially as local files, but the job is taking excessively long to initialize, and the CPU utilization on the instances remains near 0% while waiting for data. The team must optimize data loading without changing the custom PyTorch DataLoader logic. Which AWS service or feature should be used to resolve this storage performance bottleneck? - A) Mount Amazon EFS to the SageMaker instances and copy the S3 data to EFS before training. - B) Use Amazon FSx for Lustre as the input data source for the SageMaker training job. - C) Increase the instance size of the SageMaker training nodes to include larger local EBS volumes. - D) Enable S3 Transfer Acceleration on the S3 bucket.	Correct Answer: B Correct Answer Analysis: - B is Correct: Amazon FSx for Lustre integrates natively with Amazon S3. It can lazy-load files from S3 into a high-performance POSIX-compliant file system, allowing the distributed training instances to read the images concurrently at massive throughput without modifying the PyTorch file-loading logic. Distractor Analysis: - A is Incorrect: EFS is designed for general-purpose shared file systems and is not optimized for the massive throughput required by deep learning image datasets. Copying data to EFS also adds pipeline steps. - C is Incorrect: Increasing EBS volume size does not solve the network I/O bottleneck of downloading millions of small files sequentially from S3 to the instances. - D is Incorrect: S3 Transfer Acceleration is optimized for cross-region or long-distance transfers, not for high-throughput, low-latency reads from an ML training cluster within the same region. > Exam Tip / AWS Best Practice: For high-throughput ML training on large S3 datasets, use Amazon FSx for Lustre to bypass S3's object-storage latency overhead.
### Question 7 - Glue Job Out of Memory (OOM) Error An AWS Glue ETL job is ingesting and transforming a 5 TB dataset stored in Amazon S3. The job performs a large Spark `join` operation to merge user records from two different Parquet data sources. The job fails after running for 2 hours with an `Out of Memory` error on the Spark executor. The ML engineer needs to resolve this capacity issue in the most cost-effective way without changing the overall ETL logic. Which TWO actions should the ML engineer take to resolve the issue? (Select TWO) - A) Increase the number of DPUs allocated to the Glue job. - B) Enable partition pruning by filtering the data before the join. - C) Convert the Parquet data to JSON before running the Glue job. - D) Increase the driver node memory by selecting a larger worker type (e.g., G.2X). - E) Switch from AWS Glue to Amazon EMR with a 10-node cluster.	Correct Answer: A, B Correct Answer Analysis: - A is Correct: An Out of Memory error during a large Spark join often indicates insufficient executor memory. Increasing the number of DPUs scales the Spark executors, providing more aggregate memory to hold the shuffle data for the join operation. - B is Correct: If the join is processing data that is ultimately not needed, filtering the data (pushing predicates down) before the join reduces the dataset size, thus reducing the memory footprint required for the shuffle. Distractor Analysis: - C is Incorrect: Converting Parquet (a columnar, optimized format) to JSON (a row-based, inefficient format) would increase data size and worsen memory/performance issues. - D is Incorrect: While upgrading worker types (e.g., Standard to G.1X or G.2X) does increase memory, simply increasing the driver node memory does not solve executor OOM errors during a distributed join. - E is Incorrect: Migrating to Amazon EMR introduces unnecessary operational overhead and does not guarantee a more cost-effective solution compared to tuning the existing Glue job. > Exam Tip / AWS Best Practice: To fix Glue Spark OOM errors during joins, push filters down (partition pruning) to reduce data size, and increase DPUs to scale executor memory.
### Question 8 - DynamoDB Feature Lookup Hot Partition An online recommendation model uses an Amazon DynamoDB table to store real-time user features. The table is provisioned with 10,000 write capacity units (WCUs) and 10,000 read capacity units (RCUs). During a flash sale event, a specific promotional user ID generates 90% of the traffic. The application receives `ProvisionedThroughputExceededException` errors for this specific user, while overall table consumption remains at 20%. The team must resolve this immediately without changing the table's overall provisioned capacity. What is the root cause, and how can it be resolved? - A) The DynamoDB table has reached its account-level RCU limit; request a quota increase from AWS Support. - B) The write capacity is too low; double the WCUs to 20,000. - C) A hot partition is caused by a highly skewed access pattern; implement a scatter-gather pattern by appending random suffixes to the partition key. - D) The application is reading stale data; enable DynamoDB Accelerator (DAX) to cache the reads.	Correct Answer: C Correct Answer Analysis: - C is Correct: The scenario describes a classic "hot partition" issue, where a single partition key receives disproportionate traffic. DynamoDB evenly distributes throughput across partitions, but a single item/partition cannot exceed 3000 RCUs or 1000 WCUs. Appending random suffixes (e.g., 1-10) to the partition key distributes the writes/reads across multiple physical partitions, smoothing the traffic. Distractor Analysis: - A is Incorrect: The overall table consumption is only 20%, so the account or table limit is not the issue; it is the skewed distribution. - B is Incorrect: Increasing WCUs globally will not solve the issue if a single partition key is receiving all the traffic, because DynamoDB caps throughput per individual partition. - D is Incorrect: While DAX caches reads, a flash sale on a single partition key will still overwhelm the DAX cache node for that specific key, and it does not solve the write throttling. > Exam Tip / AWS Best Practice: To resolve DynamoDB hot partitions on heavily skewed keys, use the "write sharding" (scatter-gather) technique by adding random suffixes to the partition key.
### Question 9 - MSK Consumer Group Lag A machine learning data ingestion pipeline uses Amazon Managed Streaming for Apache Kafka (Amazon MSK) to collect IoT telemetry data. A consumer group running on Amazon EC2 instances processes the data and writes it to an Amazon RDS PostgreSQL database. The team observes that the `consumer lag` metric is growing continuously, but the EC2 instances have low CPU and memory utilization. The pipeline must be scaled to handle the incoming throughput. What is the MOST likely cause of the lag, and what is the most appropriate solution? - A) The EC2 instances are under-powered; upgrade to larger instance types. - B) The number of Kafka topic partitions is less than the number of EC2 consumers. - C) The number of Kafka topic partitions is equal to or less than the maximum parallelism, and the RDS database is bottlenecking the writes. - D) The MSK broker instances are too small; scale up the MSK broker instance type.	Correct Answer: C Correct Answer Analysis: - C is Correct: Low CPU/memory on the EC2 consumers indicates they are not busy processing; they are likely waiting on I/O. If the RDS database cannot write fast enough, the consumers will block waiting for DB commits, causing consumer lag to grow despite available compute resources. Distractor Analysis: - A is Incorrect: Upgrading EC2 instances will not help if they are already under-utilized (low CPU/memory). - B is Incorrect: If partitions were less than consumers, some consumers would be idle, but this still wouldn't explain low CPU on active consumers unless they were blocked downstream. The scenario strongly points to downstream I/O blocking. - D is Incorrect: MSK broker size affects broker disk/network limits and producer throughput, but consumer lag with low consumer CPU points to the processing or write layer, not the broker holding the stream. > Exam Tip / AWS Best Practice: Consumer lag combined with low consumer CPU almost always indicates a downstream bottleneck (like a database write limit) or insufficient topic partitions.
### Question 10 - OpenSearch Cluster Sharding and Indexing A search analytics team is bulk-indexing 1 billion documents into an Amazon OpenSearch Service cluster. The cluster consists of 3 data nodes. The indexing job is failing with frequent `CircuitBreakerException` and `EsRejectedExecutionException` errors. The team must resolve the indexing capacity issue while ensuring the cluster remains highly available during query execution. Which TWO actions should the team take to resolve the ingestion bottleneck? (Select TWO) - A) Increase the number of OpenSearch data nodes. - B) Increase the size of the OpenSearch EBS volumes without changing the instance count. - C) Tune the OpenSearch cluster to increase the `indices.thread_pool.index.queue_size`. - D) Temporarily disable replicas during the bulk ingestion phase. - E) Reduce the number of shards to 1 to consolidate indexing load.	Correct Answer: A, D Correct Answer Analysis: - A is Correct: `EsRejectedExecutionException` means the thread pool queue for indexing is full because the nodes cannot process data fast enough. Adding more data nodes horizontally scales the indexing capacity across more compute resources. - D is Correct: By default, OpenSearch creates replicas for high availability. During a massive bulk ingestion, indexing to replicas consumes half of the cluster's write throughput. Temporarily disabling replicas (setting `number_of_replicas` to 0) doubles the indexing speed, and they can be re-enabled after ingestion. Distractor Analysis: - B is Incorrect: Increasing EBS volume size only adds storage capacity, not compute throughput (CPU/IOPS for indexing). It will not solve thread pool exhaustion. - C is Incorrect: While increasing the queue size allows more requests to wait in memory, it increases the risk of JVM OutOfMemory errors (leading to `CircuitBreakerException`) and only delays the inevitable rejection if compute capacity isn't increased. - E is Incorrect: Reducing to 1 shard concentrates the indexing load onto a single thread on a single node, drastically reducing parallelism and worsening the bottleneck. > Exam Tip / AWS Best Practice: For massive OpenSearch bulk ingestion, scale out data nodes and temporarily set `number_of_replicas` to 0 to maximize write throughput, then restore replicas.