AWS makes building and securing data lakes easier

Amazon Web Services (AWS) has announced three new services that provide automation and prescriptive guidance to help customers build more quickly on AWS.

AWS Control Tower gives customers an automated "landing zone" that makes it easy to set up their multi-account environment and continuously govern their AWS workloads with rules for security, operations, and compliance.

AWS Security Hub is a central place to manage security and compliance across an AWS environment so that customers can quickly see their AWS security and compliance state in one comprehensive view.

AWS Lake Formation makes it easier for customers to build a secure data lake by simplifying and automating many of the complex manual steps usually required, including collecting, cleaning, and Catalogueing data.

As the cloud has increasingly become the default choice for organisations, it has attracted two distinct types of builders.

The first group are the tinkerers who want to use the full range of AWS services, pick out various building blocks, and hammer them together to create a specific architecture.

The second type of builders are willing to trade some of the service granularity to start at a higher abstraction layer where much of the construction has already been done for them and they can get started faster.

AWS caters to customers who want to build from the ground up, while also adding services over time like AWS Elastic Beanstalk, Amazon Lightsail, and Amazon SageMaker that are designed to serve the needs of this second type of builder.

AWS Control Tower, AWS Security Hub, and AWS Lake Formation extend this approach to a wider array of workloads and scenarios, giving customers abstracted services to help with provisioning and governance, monitoring security and compliance, and building and managing data lakes.

These services automate multi-step processes to get customers up and running more quickly, while also centralising management of crucial elements of their AWS environment, helping maintain consistency and providing more complete visibility.

"Our customers want us to package our services in ways that make it easier for them to build an architecture quickly," says AWS senior vice president Charlie Bell.

"One of the central benefits of the cloud is that it removes the vast operational complexities of managing physical infrastructure.

"AWS's new services abstract away additional complexity, speeding and simplifying the process of deploying and managing cloud workloads, so customers can build faster, operate more securely, and maintain consistent governance in a way that gives them more time to innovate.

AWS Control Tower makes it easy to set up and govern a secure, compliant multi-account environment

Enterprises migrating to AWS often need to manage a large number of accounts, distributed teams, and applications.

AWS's provisioning and management services give customers granular control over their environments, but many organisations also want prescriptive guidance and help setting up a secure, multi-account environment.

They want to make sure both that they're using all the right tools and that they understand how those tools can be used to create and enforce policies for their teams to deploy workloads in a secure and compliant way.

And they want all of this without sacrificing the speed and agility of AWS.

AWS Control Tower addresses these challenges by providing central cloud teams with a single, automated "landing zone" where their teams can provision accounts and workloads according to industry and AWS best practices.

The automated landing zone employs best-practices blueprints, such as configuring a multi-account structure using AWS Organisations, managing user identities and federated access with AWS Single Sign-on or Microsoft Active Directory, configuring an account factory through AWS Service Catalogue, centralising a log archive using AWS CloudTrail and AWS Config, and more.

AWS Control Tower offers pre-packaged governance rules for security, operations, and compliance, which customers can apply enterprise-wide or to groups of accounts to enforce policies or detect violations.

All of this can be easily managed and monitored through AWS Control Tower's dashboard, providing customers with centralized visibility into a customer's AWS environment including information such as accounts provisioned, preventive and detective guardrails enabled, and the compliance status of accounts as it relates to the guardrails.

AWS Security Hub provides centralised management for security and compliance 

Enterprises today use a broad array of AWS and third-party security tools that are continuously generating findings viewable in multiple consoles and dashboards.

To understand their overall security and compliance state, customers must either manually pivot between these tools or invest in developing complex systems to aggregate and analyse the findings.

As a result, many security teams find it challenging to separate the signal from the noise, prioritise the findings that matter most, and ensure that critical information isn't missed.

With AWS Security Hub, customers can quickly see their entire AWS security and compliance state in one place.

AWS Security Hub collects and aggregates findings from the security services it discovers in a customer's environment, such as intrusion detection findings from Amazon GuardDuty, vulnerability scan results from Amazon Inspector, sensitive data identifications from Amazon Macie, and findings generated by a range of security tools from AWS Partner Network (APN) partners.

Findings are correlated into integrated dashboards that visualise and summarise a customer's current security and compliance status, and highlight trends and Amazon Elastic Compute Cloud (Amazon EC2) instances that are generating an increasing number of findings.

Customers can run automated, continuous configuration and compliance checks based on industry standards and best practices, such as the Center for Internet Security (CIS) AWS Foundations Benchmark, identifying specific accounts and resources that require attention.

AWS Security Hub integrates with Amazon CloudWatch and AWS Lambda, allowing customers to execute automated remediation actions based on specific types of findings.

Customers can also integrate AWS Security Hub with their automation workflows and third-party tools like ticketing, chat, and Security Information and Event Management (SIEM) systems to quickly take action on issues.

Providers such as Alert Logic, Armor, Barracuda, Check Point, Cloud Custodian, CrowdStrike, CyberArk, Demisto, F5, Fortinet, GuardiCore, IBM, McAfee, Palo Alto Networks, Qualys, Rapid7, Splunk, Sophos, Sumo Logic, Symantec, Tenable, Trend Micro, Turbot, and Twistlock have built integrations with AWS Security Hub, with more coming soon.

AWS Lake Formation makes it easier to set up a secure data lake

A lot of customers are talking about data lakes because they understand that it is useful to remove siloes and have all of their data reside in a central place where they can apply analytics and machine learning.

Amazon Simple Storage Service (Amazon S3) is the natural place to have a data lake because of all of the security, access control, operational performance, functionality, and management controls it offers, as well as the proximity to the analytics and machine learning services available in AWS.

It turns out, however, that building and managing a data lake is a complex and time-consuming process that requires customers to load data from diverse sources, set up buckets and partitions, clean and prepare data, configure and enforce security policies across a range of services, granularly configure access control settings, and more.

AWS Lake Formation removes much of this heavy lifting, allowing customers to build a data lake in a matter of days.

Using AWS Lake Formation, customers simply define the data sources they wish to ingest and then select from a prescribed list of data access and security policies to apply, removing the need to define and enforce policies across their analytics applications.

AWS Lake Formation then collects the data and moves it into a new Amazon S3 data lake, extracting technical metadata in the process to Catalogue and organise the data for easier discovery.

The service automatically optimises the partitioning of data to improve performance and reduce costs, transforms data into formats like Apache Parquet and ORC for faster analytics, and also uses machine learning to deduplicate matching records to increase data quality.

Customers can use AWS Lake Formation to centrally define and manage security, governance, and auditing policies for their data lake.

Policies will then be consistently implemented, reducing the duplicative work of manually defining and enforcing them across a customer's security, storage, analytics, and machine learning services.

AWS Lake Formation also provides a centralised, customisable catalogue which describes available data sets and their appropriate business use.

This reduces the time that analysts and data scientists must spend hunting down the right data set for their needs, and enables users to conduct combined, self-service analytics jobs with their choice of analytics and machine learning services, including Amazon EMR, Amazon Redshift, Amazon Athena, Amazon Sagemaker, and Amazon QuickSight.