Content Domain 3: Cloud Technology and Services

Task Statement 3.6: Identify AWS storage services

Let's get started with the sixth task statement covering AWS storage services. Again, a fundamental needed is to understand what is cloud storage. Cloud storage is a cloud computing model that stores data through a cloud computing provider that manages and operates data storage as a service, giving you the agility, global scale and durability with any time, anywhere data access. So how does cloud storage work? Cloud storage is purchased from a third-party cloud vendor who owns and operates data storage capacity and delivers it over the internet in a pay-as-you go model. These cloud storage vendors manage capacity, security, and durability to make data accessible to your applications all around the world. There are benefits of cloud storage, such as no hardware or storage to provision, on-demand capacity, performance and retention, storage lifecycle management to move data to lower cost tiers and only pay for the storage you need and use. And there are cloud storage requirements such as durability, availability, and security to ensure your data is safe, secure, and available when needed. There are types of cloud storage you need to know for the exam. Object storage, file storage, and block storage. Each offers their own advantages and have their own use cases. Let's start with object storage. Object storage solutions such as Amazon S3 is a global resilient service, so it is a global storage platform. Amazon S3 is run in every AWS Region and your data is also stored in a specific Region and can be accessed from anywhere. And remember, Amazon S3 is a public service and runs from the AWS public zone. Your data stored in S3 is replicated across different Availability Zones in that Region. S3 can tolerate a failure of an Availability Zone and can also replicate data between Regions, too. It is designed to store virtually an unlimited amount of data and multiple users can access your data in S3. And keywords for S3 are unlimited amount of data. With Amazon S3, you can create buckets and add objects to your buckets. The S3 buckets hold your objects like a container. Objects can be videos, pictures, files, large datasets, and more. And the size of the object in S3 can be from 0 bytes to 5 terabytes in size. Let's mention compliance, too, because when you store objects in an S3 bucket, that S3 bucket is in a specific Region and the laws and rules of that Region also apply to your S3 bucket. When you create an S3 bucket, you must add a globally unique name for that bucket. No one bucket can have the same name as another bucket, no matter which Region or where in the world that S3 bucket sits. Here's a question. What feature of Amazon S3 can you use to ensure the objects stored in your S3 bucket are not accidentally overwritten or deleted? S3 has a lot of great features to organize and manage your data for different use cases to enforce security and meet compliance requirements. S3 features can append metadata tags to objects, move and store data across the S3 storage classes, configure and enforce data access controls, secure data against unauthorized users, run big data analytics, monitor data at the object and bucket levels, and view storage usage and activity trends across your organizations. Objects can be accessed through S3 access points or directly through the bucket host name. But to answer the question, if you enable S3 versioning during the bucket creation, every time you upload an object, it will automatically create a version of it. So if you accidentally overwrite an object, you can still use versioning to restore the previous object. For your exam, take some time to understand the different S3 storage classes and use cases. If you are asked to choose the most cost-effective storage class to store archives for a long time and need a retrieval time of less than 12 hours, which S3 storage class do you choose? Amazon S3 Glacier or Amazon S3 Glacier Deep Archive? Glacier Deep Archive is the lowest cost storage class and supports long-term retention and data can be restored within 12 hours. If you are asked to choose an object store system, then that choice should be Amazon S3, because S3 is an object store system. It is not a file system like Amazon EFS or a block storage system like Amazon EBS. Let's check out those too. File storage is another storage option in AWS. Some applications need to access shared files and require a file system. This type of storage is often supported with a Network Attached Storage server. File storage solutions such as Amazon EFS are ideal for use cases like large content repositories, development environments, media stores, or user home directories. Amazon EFS is useful with most AWS infrastructures, because it provides network-based file systems that can be mounted with Linux EC2 instances and then can be used by multiple instances at the same time. The data is stored outside of the Amazon EC2 instance, but sits inside your Amazon VPC, so it provides scaling and self-healing properties. It can be accessed using hybrid methods like VPN, Direct Connect, and VPC Peering and you can also use lifecycle policies to move data between the two different storage classes in Amazon EFS. Amazon FSx is also a file server similar to Amazon EFS. Linux is a keyword for Amazon EFS. EFS is a shared file system for Linux and you cannot use Amazon EFS for Windows. So AWS created FSx for Windows and it is a fully managed Windows file system share drive that supports the server message block protocol and Windows network technology file system. It supports active directory integration and access control list. It is built on a solid state drive, or SSD, and is a highly scalable distributed file system for Windows managed by AWS. It can scale up to 10 gigabytes per second and millions of IOPS. It is highly available and can be configured for multi-AZ and can be accessed from your on premises, and your data is backed up daily to S3. There is also Amazon FSx for Lustre, and Lustre is a type of parallel distributed file system for large scale computing. Lustre's name comes from Linux and cluster, so that will help you remember it. It is for Linux instances and cluster is for large-scale computing. For the exam, know that Lustre is great for machine learning and high-performance computing and has a sub-millisecond latency. Lustre is also great for file systems to perform video processing, financial modeling, design automation, whatever needs a high level of distribution. And then there is also another option, which is block storage. Some enterprise applications such as databases often require dedicated low-latency storage for each host. This is comparable to direct-attached storage or Storage Area Network. Block-based cloud storage solutions such as Amazon EBS are provisioned with each virtual server and offered the ultra low latency required for high-performance workloads. In an earlier lesson, we mentioned that Amazon EC2 has local storage included with the AMI and this storage is called direct-attached or local-attached storage. These are physical disks that are directly connected to a device. This storage is directly connected to the Amazon EC2 host and it is called Instance Store and this type of storage is really fast, because it's attached to the hardware. But it does have a lot of issues. If the disk fails, if the hardware fails, if the instance moves between two different EC2 hosts, the storage can be lost. So we have an alternate type of storage we can use with our EC2 instances and that is Network Attached Storage and this is where volumes are created and attached to the EC2 instance over the network and AWS uses Amazon EBS for this. EBS is not as fast as Instance Store, but EBS is highly resilient and is separate from the instance hardware. So issues with the EC2 host will not impact EBS. You may also hear the term ephemeral storage and this means storage that is temporary. You cannot rely on this storage to be persistent. Instance Store is an example of ephemeral storage and EBS is an example of persistent storage. EBS is a volume that is presented to the operating system as a collection of blocks, but there is no structure, just a collection of addressable blocks And these blocks can be mounted, which means that file systems can be created on top of the block storage and you can also boot off of an EBS volume and this is why most EC2 instances use an EBS volume, block storage, as their boot volume and it is what stores the operating system. Here's a question. What type of EBS volume is recommended for most workloads and can also be used as a boot volume? Amazon EBS provides several volume types that differ in performance and price. The answer would be general purpose SSD, because it is recommended for most workloads. It can be used as a system boot volume and it is best for development and test environments. But there is also Provisioned IOPS SSD, which is meant for critical business applications that require sustained IOPS performance. It is best used for large database workloads. And there is Throughput Optimized HDD, which is meant for streaming workloads, requiring consistent, fast throughput at a low price. Big data, data warehouses, and log processing are great for this, but it cannot be a boot volume. And finally, Cold HDD, which is meant for throughput oriented storage for large volumes of data that are infrequently accessed or in scenarios where the lowest storage cost is important. It cannot be a boot volume too. You can also use EBS snapshots to create a point-in-time backup. Let's also talk about the cached file system for AWS, which is AWS Storage Gateway. AWS Storage Gateway is a service that allows us to connect our on-premises data center storage to an AWS storage service and it helps to migrate part or all of your storage platform to AWS or it supports extending your storage platform to AWS. Storage Gateway is a virtual appliance that you run on your on-premises virtualization platform, so you can run it on VMware, ESXi, EC2, and so on. So Storage Gateway is a product you download and configure on your on-premises location and it talks to AWS over the internet. There are three types of storage gateway that are available. Storage Gateway File Gateway that stores files as objects in S3, but has a local cache for your most recently accessed data on site. File gateway is a great option for Windows and is accessible using the SME protocol and when you upload data with your Storage Gateway, that data is stored in S3. Volume gateway is configured the same way as file gateway. So you configure and download the virtual machine image, but when you access it, you are not accessing file shares, but volumes and these volumes are accessed over the protocol iSCSI, which stands for Internet Small Computer Systems Interface and it is an internet protocol-based storage networking standard for linking data storage facilities. iSCSI provides block level access to storage devices by carrying SCSI commands over TCP/IP network and are generally used with SAN products. So it is Network Attached Storage and volume gateway works great with servers. And you have two options for volume gateway. Gateway Stored Volumes to store all your data on a volume gateway appliance itself and then snapshots are taken into AWS and stored on S3. It's great to use when you want your data on-premises, but snapshots and backups in S3. And then there is also Gateway Cached Volume to store primary copies of your data in AWS and then downloads and caches frequently accessed data on the volume gateway itself. The last type of storage gateway is the Virtual Tape Library Gateway to present a virtual tape library over iSCSI to any compatible backup software. This has a very high administrative overhead and is also costly and should be stored off site for disaster recovery and best practices. It allows us to present a virtual tape drive and it is stored in S3 and you can use this for your migration for your data into AWS over a period of time. Let's wrap up this lesson and talk about storage backups. Backup and recovery is critical to ensure your data is protected and accessible, but keeping up with increasing capacity requirements can be a constant challenge. Cloud storage brings low cost, high durability, and extreme scale to backup and recovery solutions. Amazon S3 offers a full range of storage classes, which offers trade-offs between price, durability, performance, and speed of access. Each storage class offers different cost, performance, retrieval speed, resilience, and so on, but it is a great choice and a low-cost storage you can use for backups. Embedded data management policies such as S3 Object Lifecycle Management can automatically migrate data to lower cost tiers based on frequency or time settings. And archive vaults such as Glacier and Glacier Deep Archive can be created to help comply with legal or regulatory requirements. These benefits allow for tremendous scale possibilities within industries such as financial services, healthcare and media that produce high volumes of data with long-term retention needs. There's also AWS Backup, which is a fully managed service that helps to centralize and automate data protection and meet compliance requirements across AWS services and on premises. Using this service, you can configure backup policies and monitor activity for your AWS resources in one place. It allows you to automate and consolidate backup tasks that were previously performed service by service and it removes the need to create custom scripts and manual processes. The AWS storage services vary for different use cases, functionality, optimizations, management, and more. Ensure you take some time to understand how each of these services work, not only how the services operate in their limitations, but also look into their actual usage. The AWS storage services vary in their use cases, functionality, optimization, management, and more and you should take some time to understand how each of these services work and how they communicate, how they might work with each other and of course, how they fit into the types of architectures you will build. Let's get started with the seventh task statement.