AWS Serverless with Terraform

Architecture | Vlad Cenan |
10 December 2019

After building and managing an AWS Serverless Infrastructure using Terraform over the last 7 months, I want to share some best practices and some common mistakes that I faced during this workflow.

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.

Terraform's purpose on this project was to provide and maintain one workflow to provision our AWS Serverless Stack infrastructure. The following is a collection of lessons that I have personally learned, and a few tricks and workarounds that I have come across to get Terraform to behave the way I wanted.

What is Serverless?

Serverless computing is a cloud computing model in which a cloud provider automatically manages the provisioning and allocation of compute resources.

This implementation of serverless architecture is called Functions as a Service (FaaS).

Why Serverless?

Using serverless had a lot of advantages and benefits, especially when the client is focused on cutting costs as they only pay for the execution and duration, and do not have to maintain a server, operating system or installation. Also, serverless offers automated high availability, that reduces the time spent on architecting and configuring these capabilities.

Some disadvantages that we faced included I/O bottlenecks that we couldn’t replicate and the lack of visibility in debugging our application flows.

Best Practice

Security

Hardcoded passwords are not only dangerous because of the threat of hackers and malware, but they also prevent code reusability.

AWS Secret Manager is a great service for managing secrets, storing, retrieving and rotating shared secret keys between resources.

Please heed this advice and store your secrets and keys in a secret manager tool not on a laptop or hardcoded in git.

Versioning

To create a lambda function, the deployment package needs to be a .zip consisting of your code and any dependencies. The archives are uploaded in an S3 bucket based on a timestamp.

In my case, the timestamp will be the version for lambda functions and the key used by terraform to deploy the proper lambda function:

vcenan@devops:~$ aws s3 ls s3://bucket/lambda-function/

PRE v2019-03-01345

To track the latest builds, a manifest file was added to the project which is constantly updated with every build and tagged based on releases.

From a security perspective, I would recommend S3 Server-Side Encryption, in order to protect sensitive data at rest. If you transfer data to S3, it is TLS encrypted by default.

Resources

Adopt a microservice strategy, and store terraform code for each component in separate folders or configuration files.

Instead of setting all dependencies for a resource into one configuration file, break it down into smaller components.

In the example below, the lambda function resource will take the IAM role from another terraform configuration file iam.tf (file responsible with creating all the roles for AWS resources) and will get the role definition from a .json file:

vcenan@devops:~$ cat lambda.tf

resource "aws_lambda_function" "example" {

function_name = "${var.environment}-${var.project}"

s3_bucket = "${var.s3bucket}"

s3_key = "v${var.app_version}/${var.s3key}"

handler = "main.handler"

runtime = "python3.7"

role = "${aws_iam_role.lambda_exec.arn}"

}

vcenan@devops:~$ cat iam.tf

# IAM role which dictates what other AWS services the Lambda function may access.

resource "aws_iam_role" "lambda_exec" {

name = "${var.environment}_${var.project}"

assume_role_policy = "${file("iam_role_lambda.json")}"

}

vcenan@devops:~$ cat iam_role_lambda.json

{

"Version": "2012-10-17",

"Statement": [{

"Action": "sts:AssumeRole",

"Principal": { "Service": "lambda.amazonaws.com" },

"Effect": "Allow",

"Sid": "" } ]

}

This will help you in debugging and re-using components and of course for a better visibility.

Passing Variables

Just like any tool or language, Terraform supports variables. All of the typical data types are supported which makes them really useful and flexible. An input variable can be used to populate configuration input for resources or even determine the value of other variables.

In the example below we are getting the remote state (which holds the resources and metadata for the created infrastructure) from the flow that was deployed and map it in our current flow that will deploy the triggers.

vcenan@devops:~$ cat outputs.tf

# Get Remote State

data "terraform_remote_state" "distribution" {

backend = "s3"

config {

bucket = "s3bucket-name"

region = "eu-west-2"

key = "terraformState/dev/distribution.tfstate"

}

vcenan@devops:~$ cat s3notification.tf

# S3 Notification

resource "aws_s3_bucket_notification" "event_notification" {

bucket = "${var.s3_store}"

lambda_function {

lambda_function_arn = "${data.terraform_remote_state.distribution.distribution-lambda-function_arn}"

events = ["s3:ObjectCreated:Put"]

filter_prefix = "${var.s3_event_ distribution}"

}

…

See more ˅

By defining the output for the target lambda, we can reference the ARN (resource name) of the lambda created in this module.

vcenan@devops:~$ cat outputs.tf

output “distribution-lambda-function-id” {

value = “${aws_lambda_function.distribution-lambda-function.id}”

}

If you want a module output or a resource attribute to be accessible via a remote state, you must thread the output through to a root output.

module "app" {

source = "..."

}

output "app_value" {

value = "${module.app.value}"

}

Reusability

When it gets to more complex architectures like having multiple environments with same resources, things can get unwieldy. In order to avoid copying files between environments, which leads to redundancy, inconsistency, and inefficiency, use terraform modules.

Terraform’s way of creating modules is very simple: create a directory that holds a selection of .tf files. That module can be called in each of the environment modules.

vcenan@devops:~$ cat elasticache/main.tf

resource "aws_elasticache_replication_group" "elasticache-cluster" {

availability_zones = ["us-west-2a", "us-west-2b"]

replication_group_id = "tf-rep-group"

node_type = var.node_type

number_cache_clusters = var.number_cache_cluster

Parameter_group_name = "default.redis3.2"

port = 6379

}

vcenan@devops:~$ cat environments/dev/main.tf

module "dev-elasticache" {

source = "../../elasticache"

number_cache_clusters = 1

node_type = "cache.m3.medium"

}

This was we can also make your modules configurable in case we need different parameters in production environment.

vcenan@devops:~$ cat environments/prd/main.tf

module "prd-elasticache" {

source = "../../elasticache"

number_cache_clusters = 3

node_type = "cache.m3.large"

}

API Gateway

For a better view, use Swagger to define your API Gateway to:

keep your Terraform code more concise;
have a clear overview of the API definition with an online Swagger editor;

This can be done simply with aws_api_gateway_rest_api terraform resource which will reference the body of the swagger file. The template_file terraform resource will allow you to fill the swagger file body with other terraform resources or outputs and render at runtime.

Unit Tests

A good way to test your infrastructure is to use awsspec ruby gem, a plugin that will test your AWS resources.

In the example below, unit tests are successfully passed for our AWS Dynamo DB deployed:

dynamodb_table ‘metadata’

should exists

should be active

should have key schema “ProcessName”

provisioned_throughput.read_capacity_units

should eq 5

provisioned_throughput.write_capacity_units

should eq 5

item_count

should eq 29

A disadvantage is that not all AWS resources are covered, like Athena DB or Step Functions, which means they become time-consuming to develop.

Debugging

Terraform has a detailed log which can be enabled by using the environment variable TF_LOG:

export TF_LOG=DEBUG

export TF_LOG_PATH=./terraform.log

You can use more log levels and, in this example terraform saves the debug logs from the session in the current location in terraform.log file.

Terraform State

Terraform must store state about your managed infrastructure and configuration. Backends are responsible for storing the state and providing an API for state locking.

If you want to use Terraform as a team on a product, you will need to enable the state locking feature in backend that prevents multiple runs for terraform components.

remote_state {

backend = "s3"

config = {

bucket = "s3-bucket"

key = "${path_relative_to_include()}/terraform.tfstate"

region = "eu-west-1"

encrypt = true

dynamodb_table = "locks"

}

Vlad Cenan

DevOps Engineer

Vlad is a DevOps engineer with close to a decade of experience across release and systems engineering. He loves Linux, open source, sharing his knowledge and using his troubleshooting superpowers to drive organisational development and system optimisation. And running, he loves running too.

All Categories

From This Author

25 February 2019

Infrastructure as Code with Terraform