Skip to content

Skill 1.1.9: Ingest data into SageMaker Feature Store

Task 1.1: Data Preparation for ML and AI — Collect and store data

Core objective: one governed, reusable feature definition that serves both training (offline, historical, point-in-time) and inference (online, low-latency, latest). The business win is eliminating training–serving skew and duplicated feature pipelines.

Feature Group Anatomy

A Feature Group is the Feature Store resource. Every record = RecordIdentifier + EventTime + feature values.

Element Rule
RecordIdentifierFeatureName Required. Must be String or Integral. Unique entity key (e.g. customer_id).
EventTimeFeatureName Required. String (ISO-8601) or Fractional (Unix epoch seconds). Prefer Fractional.
FeatureDefinitions Name + type. Types: String, Integral, Fractional, plus CollectionType (List/Set/Vector) for embeddings. No Boolean — encode as String.
Naming Unique per account/Region; ≤64 chars. Glue table = lowercase name, - → _.
Default Glue database sagemaker_featurestore

Gotcha: Feature name and type are immutable after creation. Schema evolution = add features via UpdateFeatureGroup, or create a new feature group.

Online vs Offline Store

Online Store Offline Store
Purpose Real-time inference lookups Training, batch scoring, analytics
Storage Managed low-latency store (not S3, not in your VPC) Amazon S3 + AWS Glue Data Catalog
Format — Apache Parquet (default) or Apache Iceberg
Read path GetRecord, BatchGetRecord — only Athena / Spark / Glue over the Glue table
Contents One record per record ID (latest EventTime wins) Append-only history of every version
Write latency Milliseconds Minutes after PutRecord (streamed asynchronously)
Enable at create OnlineStoreConfig.EnableOnlineStore OfflineStoreConfig (optional)
Add later? No — cannot add an online store after creation Yes — UpdateFeatureGroup can add one
Retention TtlDuration (default 0 = never expires) S3 lifecycle policy (you manage)
Encryption OnlineStoreConfig.SecurityConfig.KmsKeyId OfflineStoreConfig.S3StorageConfig.KmsKeyId

Gotcha: Athena cannot query the online store. Conversely, GetRecord never returns history — use the offline store for time travel.

Gotcha: Online-only and offline-only feature groups are both legal. Online-only = no history, no point-in-time joins. Offline-only (EnableOnlineStore: false) = cheaper, batch-only, no GetRecord.

Ingestion Paths — Choose by Latency & Target

Path Mechanism Writes Online Writes Offline
Real-time / streaming PutRecord (single, synchronous) ✅ immediately ✅ async (~minutes)
High-volume near-real-time BatchWriteRecord ✅ ✅
Bulk backfill / historical load Athena INSERT INTO or Spark connector writeToFeatureStore ❌ No ✅ only
Stream processing Kinesis Data Streams / MSK → Lambda or Managed Service for Apache Flink → PutRecord ✅ ✅
No-code transform SageMaker Data Wrangler export to Feature Store ✅ ✅

Gotcha: A bulk backfill via Athena/Spark populates the offline store only. The online store stays empty until those records are also pushed with PutRecord. This is the #1 trap for this skill.

Gotcha: PutRecord validates strictly against FeatureDefinitions. Missing record ID, missing event time, unknown feature name, or a type mismatch → ValidationException. A String value for an Integral feature fails.

Point-in-Time Correctness (why Feature Store exists)

  • Point-in-time join: for each label at time T, join feature values as of T, not the latest values.
  • Prevents label leakage / target leakage → inflated offline metrics, disappointing production results.
  • Implement with Athena SQL (window functions over the offline table), the SageMaker Feature Store Spark connector, or the legacy create_dataset API's point_in_time_accuracy parameter.
  • Training–serving skew is removed only when training and inference read the same feature group definition — never hand-roll a separate training query against raw S3.

Security, IAM & Network

  • Data-plane IAM: sagemaker:PutRecord, sagemaker:BatchWriteRecord, sagemaker:GetRecord, sagemaker:BatchGetRecord, sagemaker:DeleteRecord. Control plane: CreateFeatureGroup, UpdateFeatureGroup, DeleteFeatureGroup.
  • Execution role (RoleArn) needs S3 read/write on the offline prefix, KMS Encrypt/Decrypt on both CMKs, and AWS Glue permissions if Feature Store creates the catalog table (DisableGlueTableCreation to opt out).
  • Encryption at rest: separate customer-managed KMS keys for online and offline. Default is the AWS-managed aws/sagemaker key — not acceptable for strict compliance.
  • VPC: the online store is a managed service outside your VPC. Use the interface endpoint com.amazonaws.<region>.sagemaker.featurestore-runtime for PutRecord/GetRecord, plus sagemaker.api for the control plane.
  • Cross-account: offline store in another account's S3 bucket is supported (bucket policy + KMS key policy). Cross-Region is not supported. The online store is same-account only.

Deletion Semantics

API Effect
DeleteRecord + SoftDelete (default) Removes from online store; writes a delete marker in the offline store (data remains in S3)
DeleteRecord + HardDelete Physically removes from both online and offline
DeleteFeatureGroup Deletes online store data. Offline S3 data is not deleted — clean up separately.

Gotcha: DeleteRecord requires the record's EventTime. There is no "delete by ID across all history" in one call.

Throughput, Capacity & Troubleshooting

  • Throughput modes: OnDemand (default) vs Provisioned (ProvisionedReadCapacityUnits / ProvisionedWriteCapacityUnits). Provision for predictable cost at sustained high TPS.
  • ThrottlingException on PutRecord → online store capacity exceeded. Switch to on-demand, raise provisioned capacity, or batch with BatchWriteRecord.
  • Record never appears in the offline store → offline store not enabled, or still inside the async streaming window. Not a failure.
  • Backfilled records absent from GetRecord → the backfill wrote offline only.
  • ResourceNotFound → feature group not in Created state (check Creating / CreatingFailed).
  • Out-of-order events: an older EventTime will not overwrite the newer online value, but it is appended to offline history.
  • Lagging stream consumers → bottleneck is almost always the downstream write target, not consumer count. Scale the target first.

Anti-Patterns (common distractors)

  • ❌ Querying the online store with Athena/Spark → use GetRecord / BatchGetRecord.
  • ❌ Building the training set from the online store → no history; use the offline store with a point-in-time join.
  • ❌ Writing Parquet directly into the offline S3 prefix to "backfill faster" → bypasses the catalog and online store, can corrupt the Glue table. Use Athena INSERT INTO or the Spark connector.
  • ❌ Treating Feature Store as a general-purpose database → it is not a query engine for the online store; pair with DynamoDB/RDS for arbitrary queries.
  • ❌ Expecting vector similarity search from Feature Store → that is OpenSearch Service, RDS pgvector, or S3 Vectors. Feature Store holds feature values, not a nearest-neighbor index.
  • ❌ Using PutRecord for bulk ingestion → it is single-record; use BatchWriteRecord or the offline bulk path.