Skip to content

Skill 1.1.3: Troubleshoot and debug data ingestion and storage issues that involve capacity and scalability

Task 1.1: Data Preparation for ML and AI - Collect and store data

Objective: Keep ingestion/storage within service quotas while meeting throughput/latency. Capacity = hard limits; scalability = design cannot grow linearly. Find the narrowest bottleneck: source → network → ingestion → processing → sink.

Diagnostic Matrix

Symptom Likely root cause First checks Fix
Consumer lag grows Ingress > processing or downstream sink saturated Kinesis shard throughput, MSK partitions, Flink parallelism, sink IOPS Add shards/partitions, scale consumers, scale sink, compress/batch
Throttling (429/ProvisionedThroughputExceeded) Exceeded WCU/RCU, KMS, S3 request rate, API quota CloudWatch ThrottledRequests, hot partition metrics On-demand, partition key redesign, retry/backoff, raise provisioned
S3 slow uploads Single-stream PUT, many small files, prefix hotspot Multipart upload, prefix distribution, Transfer Acceleration Multipart, parallel uploads, random prefixes, compact files
EBS latency/IOPS Volume type/limit, burst depletion VolumeQueueLength, BurstBalance gp3 provisioned IOPS/throughput, io2, RAID, instance EBS limits
EFS slow Burst credits, throughput mode BurstCreditBalance, PercentIOLimit Elastic/Provisioned throughput, larger size
RDS slow writes Storage IOPS, max connections, replication lag WriteIOPS, DBLoad, connection count Storage autoscaling, io2, read replicas, RDS Proxy
DynamoDB hot partition Poor partition key, uneven access ConsumedCapacity, ThrottledRequests On-demand, write sharding, adaptive capacity, DAX
OpenSearch indexing slow Shard count, bulk size, refresh interval, JVM ThreadPoolRejected, CPU, JVM pressure Add shards/replicas, bulk tuning, refresh interval, instance types
Feature Store online latency TPS capacity, hot key Online store metrics Provisioned capacity, better key design, offline batch

Streaming Ingestion Capacity

  • Kinesis Data Streams: shard = 1 MB/s ingress, 2 MB/s egress, 1000 records/s. Consumer parallelism ≤ shards. Enhanced fan-out = 2 MB/s per consumer. On-demand auto-scales; provisioned requires manual shard scaling.
  • MSK / Kafka: partition = unit of parallelism. Consumer group max = partitions. Adding partitions increases throughput but breaks key ordering across partitions.
  • Managed Flink: parallelism, checkpointing, state backend, backpressure. Scale parallelism or operator parallelism.
  • Troubleshoot: consumer lag grows? Check if downstream sink is saturated. Adding consumers to a blocked sink moves the queue. If ingress > shard limit, add shards/partitions or compress/batch.

Exam gotcha: Kinesis shard egress is shared across consumers. Adding consumers beyond shard count does not increase read throughput unless enhanced fan-out is enabled.

Storage Capacity & Scaling

Service Limit / scaling unit Scaling lever Anti-pattern
S3 3,500 PUT/5,500 GET per prefix/s; unlimited storage Multipart upload, parallel requests, random prefixes, Transfer Acceleration Many small files, single prefix hotspot, sequential upload
EBS Volume size/IOPS/throughput; gp2 burst credits gp3 provisioned IOPS/throughput, io2, RAID, instance EBS bandwidth Relying on burst for sustained load
EFS Bursting tied to size; Provisioned/Elastic throughput Elastic throughput, Provisioned throughput, larger size Bursting mode for high sustained throughput
FSx for Lustre Throughput scales with capacity; SSD vs HDD Increase capacity, multiple file systems Underprovisioning for training jobs
RDS Storage size, IOPS, max connections Storage autoscaling, io2, read replicas, RDS Proxy Single instance for read-heavy feature extraction
DynamoDB 3,000 RCU / 1,000 WCU per partition; 400 KB item On-demand, partition key design, write sharding, DAX Hot partition, scan instead of query
OpenSearch Shard count, JVM heap, disk Add shards/replicas, instance types, index rotation Too many small shards, large bulk requests, frequent refresh
pgvector Memory, index build, work_mem HNSW/IVFFlat index, adequate RAM, read replicas Brute-force search on large embeddings
Feature Store Online TPS, offline S3 Provisioned online capacity, batch ingestion High-cardinality hot keys, unbounded online TTL

Batch ETL & Merging

  • Glue: DPU capacity. Scale workers (G.1X, G.2X, G.4X), enable auto-scaling, partition data, use Parquet/ORC, push down predicates.
  • EMR: Cluster size, instance groups/fleets, auto-scaling, EBS volumes, Spot for task nodes. Use partition pruning, columnar formats, broadcast joins.
  • Merge/Join: Skew causes stragglers. Salting, broadcast joins, partition pruning. S3 sink: compact many small files.
  • Anti-pattern: Scaling compute before fixing data layout/format.

Exam gotcha: A batch job that slows as dataset grows usually reads more data than needed. Fix layout/format (Parquet/ORC, partitioning) before adding compute.

Vector Store & Feature Store Capacity

  • Vector store selection:
  • OpenSearch: managed vector search, HNSW, scales with shards/replicas.
  • RDS PostgreSQL pgvector: embeddings beside relational data; scale vertically/read replicas; index type HNSW/IVFFlat.
  • S3 vector buckets: large, cost-effective, higher latency; good for infrequent queries.
  • Capacity troubleshooting:
  • OpenSearch: shard count, JVM heap, segment merges, refresh interval.
  • pgvector: memory, shared_buffers, work_mem, index build maintenance_work_mem.
  • Feature Store: online store for low-latency; offline for training. Provisioned online TPS. Hot keys cause throttling.
  • Training-serving skew: one feature definition across training/inference.

Security, Quotas, Observability

  • KMS throttling: KMS request quotas can bottleneck encrypted S3/EBS/RDS/DynamoDB. S3 Bucket Keys reduce KMS calls.
  • VPC endpoints: Gateway for S3/DynamoDB; Interface for Kinesis/Glue. Endpoint/ENI limits affect scalability.
  • IAM/API throttling: retries with exponential backoff.
  • Observability: CloudWatch (ConsumerLag, ThrottledRequests, VolumeQueueLength, BurstCreditBalance, WriteIOPS, JVMHeapPressure); CloudTrail; X-Ray.
  • Quotas: per-region/account; request increases proactively.

Exam gotcha: Encryption does not inherently limit scalability, but KMS API quotas and VPC endpoint bandwidth can become hidden bottlenecks.

Decision Matrix: Scale Up vs Out vs Redesign

Problem Scale up Scale out Redesign
Ingestion > processing Larger instance More consumers/shards/partitions Compress/batch, change format
Hot partition Higher provisioned Add partitions Better partition key, write sharding
S3 slow reads N/A Parallel requests Compact files, columnar format
EBS IOPS limit Larger/io2 volume RAID, multiple volumes Move to FSx for Lustre/EFS
RDS write bottleneck Larger instance, io2 Read replicas (reads), sharding Aurora, DynamoDB, caching
OpenSearch indexing Larger instances Add shards/replicas Bulk tuning, refresh interval, index rotation
Feature Store online throttling Provisioned capacity N/A Better key design, batch offline

Anti-Patterns & Exam Traps

  • Adding consumers when downstream is bottleneck: moves queue, doesn't fix.
  • Ignoring per-partition/shard limits: provisioned table capacity does not prevent hot partition throttling.
  • Row formats for analytics: CSV/JSON scan all columns; use Parquet/ORC.
  • Many small S3 files: slows query engines; compact.
  • Burst credits for sustained load: gp2/EFS Bursting deplete; use provisioned/elastic.
  • Scaling compute before fixing data layout: wastes money.
  • Assuming S3 is always best for vectors: S3 vector buckets have higher latency; OpenSearch/pgvector for low-latency.
  • Confusing feature store and vector store: Feature = identity/current+historical; vector = similarity.
  • Forgetting KMS/VPC quotas: can throttle encrypted data paths.

Final hard rule: Troubleshoot capacity by finding the narrowest bottleneck first. Scale the bottleneck, not the symptom.