- Terraform by HashiCorp
- Learn about provisioning infrastructure with Terraform
- AWS Architecture Center
- Github: hashicorp/terraform
- Terraform Module Registry
- AWS - Managing Access Keys for IAM Users
- Github jweissig/episode-57
In this episode, we will be checking out infrastructure automation with Terraform. We will look at what it is, why you might want to use it, and then walk through a couple examples.
Before we dive in, I wanted to thank you for getting a subscription!
Terraform is an open-source project, that has a well established reputation as being one of the leaders in infrastructure automation. It also has a pretty active community too, both in people adding new features, along with pretty good documentation and a rapid release cycle. This comes in handy if you get stuck, in that if you Google around, you will likely quickly find some examples and a solution. There is an Enterprise offering too, with support, and some added features built around team collaboration.
It probably makes sense to back up for a minute, and give a quick high-level introduction to the concept of infrastructure automation, along with the problem Terraform is trying to solve.
There is this pretty cool site, called the AWS Architecture Center, where they give you a bunch of Reference Architectures for various workloads running in AWS. If you scroll down a little, they have some patterns for things like Web Hosting, Batch Processing, and Fault Tolerance. If you have never seen this before, you can learn lots, just by reading these. Lets check out the web application hosting one here as I want to use this as an example for a minute.
Say you want to deploy this Reference Architecture. How do you actually do it? Well, maybe you log into the web based management console, and start using the web interface to create things like the frontend web servers, then the application servers, and wire these up to your database. Next you add load balancing, etc. You get the idea. You just start working your way through it manually.
I just wanted to comment here before we dig deeper. That, I actually do not see anything wrong with using the web interface for doing this type of stuff manually. It is pretty easy, fast, and gets things wired up really quickly. I use the web interface all the time. But, you could also used the AWS command line tools, say for example that you already know what you want to do, or you have created a set of document steps for yourself, commands that you run for spinning up this architecture. Both these are great options that I use myself.
But, there is a third option here too, besides the web interface and command line. This is where we get into infrastructure automation tools like Terraform. Basically, Terraform is a tool, that allows you to define what an architecture looks like, via a set of configuration files, and then you ask Terraform to go build it for you. To add context to the types of tools Terraform integrates with, lets quickly check out the docs for a minute. Under, Providers here, you can see Terraform works with a ton of folks, like AWS, Google Cloud, Azure, VMware, Digital Ocean, Github, etc. It is pretty crazy. Just scroll through here you will totally find tools you use.
Okay, so it probably makes sense to just show you a live example, as that will likely flush things out. Lets switch over to AWS and run through and example using them. So, I am logged into the AWS web console here. I am using the Oregon region which is us-west-2. What we are going to do it spin up an EC2 instance using Terraform from the command line. There are no EC2 instances running right now in this region.
Just a heads up here, I have created a Terraform AWS user, by going into My Security Credentials. I gave this user the AmazonEC2FullAccess permission. I also copied the API access, and secret keys, for that Terraform user for use in our examples. This is what will allow us to remotely control AWS using Terraform. I have put a link in the episode notes below how you can do this too.
In my editor, I have a Terraform config file loaded here, this format is called HashiCorp Configuration Language (or just HCL). You can see it is pretty simple, we have a block where we define the provider as AWS, and then put in our newly created Terraform users access and secret keys. I just blanked mine out here with sample values. I also specified the AWS region as us-west-2 or Oregon. This next block here, defines that I want to create an EC2 instance, web1, using this AMI and Instance Type.
Lets jump over to the command line. You can see my two examples here, we will cover both these today, and I was just showing you example 1. Before we do anything, I am going to download Terraform. It supports Mac, Windows, Linux, and a handful of others too. You can easily find the download links on their website. I will just unzip this. Then we have this statically linked terraform binary. Typically, you will see people copying this binary to a user or system bin directory, so they have access to it in their PATH. I am just going to run this locally for now.
$ ls -l -rw-r--r-- 1 jweissig staff 839 19 Feb 06:05 README.md drwxr-xr-x 3 jweissig staff 96 19 Feb 05:42 example_1 drwxr-xr-x 3 jweissig staff 96 19 Feb 05:42 example_2
$ curl https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_darwin_amd64.zip -o terraform_0.11.11_darwin_amd64.zip
$ unzip terraform_0.11.11_darwin_amd64.zip
$ ls -l -rw-r--r-- 1 jweissig staff 839 19 Feb 06:05 README.md drwxr-xr-x 3 jweissig staff 96 19 Feb 05:42 example_1 drwxr-xr-x 3 jweissig staff 96 19 Feb 05:42 example_2 -rwxrwxr-x 1 jweissig staff 106576920 14 Dec 13:20 terraform -rw-r--r-- 1 jweissig staff 22431061 19 Feb 06:50 terraform_0.11.11_darwin_amd64.zip
You can see the version is, 0.11.11, but everything we cover today should apply down the road too. We can verify this is an executable file and not some script. I like to use the file command whenever I am curious about file types. Then, if we just run terraform without any options, it will print out the help menu.
$ terraform version Terraform v0.11.11
$ file ./terraform ./terraform: Mach-O 64-bit executable x86_64
There are a handful of command options you will likely use on a daily basis (if you get going with terraform). These are init, this will download dependencies seen in your code, things like modules to support various functionality. There is the the plan option, this will parse your terraform files, do a read only check on the current state of AWS, and then tell you what it will do. You can think of this like a dry run without actually doing anything.
Next, there is apply, this will takes your terraform configuration and apply the changes to your infrastructure. So, in our case, we wanted to create a EC2 instance. What is cool about this, is that we can run the same script over and over, and terraform is smart enough to track its own changes via a state file. So, it will not create duplicates, only what we asked for, which is pretty nice.
Next, we have destroy. This is used to delete infrastructure we created with terraform. Finally, there is this show command, which allows you to inspect the state of your terraform environment.
$ terraform Usage: terraform [-version] [-help][args] The available commands for execution are listed below. The most common, useful commands are shown first, followed by less common or more advanced commands. If you're just getting started with Terraform, stick with the common commands. For the other commands, please read the help and docs before usage. Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Workspace management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a Terraform working directory output Read an output from a state file plan Generate and show an execution plan providers Prints a tree of the providers used in the configuration push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version workspace Workspace management All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
Next, lets go into the example_1 directory. In here we have our main.tf file, the tf extension, as you likely guessed, defines Terraform files. I should also mention, that in another window here, I updated these access and secret string tokens for the AWS Terraform user. I just did not want to show them here as they are secret.
$ ls -l -rw-r--r-- 1 jweissig staff 839 19 Feb 06:05 README.md drwxr-xr-x 3 jweissig staff 96 19 Feb 05:42 example_1 drwxr-xr-x 3 jweissig staff 96 19 Feb 05:42 example_2 -rwxrwxr-x 1 jweissig staff 106576920 14 Dec 13:20 terraform -rw-r--r-- 1 jweissig staff 22431061 19 Feb 06:50 terraform_0.11.11_darwin_amd64.zip $ cd example_1 $ ls -l -rw-r--r-- 1 jweissig staff 416 19 Feb 06:14 main.tf
Our goal here is to create this AWS EC2 instance. Lets verify we have no instances running by checking the AWS console again. Yup, no EC2 instances running here in us-west-2, so lets change that.
// configure API access (user with AmazonEC2FullAccess)
provider "aws" {
access_key = "ACCESS_KEY_HERE"
secret_key = "SECRET_KEY_HERE"
region = "us-west-2"
}
// Amazon Linux 2 AMI (HVM), SSD Volume, 64-bit x86
// https://www.terraform.io/docs/providers/aws/r/instance.html
resource "aws_instance" "web1" {
ami = "ami-032509850cf9ee54e"
instance_type = "t1.micro"
tags {
Name = "web1-tf"
}
}
When running Terraform for the first time, you want to run terraform init, this will parse the .tf files in our current directory, looking for and dependencies it needs to download. For example, we will need an aws module here, that knows how to connect to aws using our keys, to create our instance. As you can see here, terraform downloaded a provider plugin for aws, version 1.59.
$ terraform init Initializing provider plugins... - Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "aws" (1.59.0)... The following providers do not have any version constraints in configuration, so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below. * provider.aws: version = "~> 1.59" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
Worth mentioning, that the aws plugin is downloaded into a .terraform directory. You do not need this for anything, but I was just wondering where it went, so that is where just in case you were wondering.
$ ls -la drwxr-xr-x 3 jweissig staff 96 19 Feb 07:07 .terraform -rw-r--r-- 1 jweissig staff 446 19 Feb 07:06 main.tf
Next, lets run, terraform plan. This will parse our main.tf file, along with connecting to AWS for gathering the current state, and then terraform will tell us what is thinks needs to happen.
One cool thing about Terraform, is that it gives you these coloured symbols that help you quickly visually figure out what is happening. You can see this green plus sign, that means we are going to create something. Then we get into the actions terraform wants to take, which is creating our AWS instance, using the AMI and instance type we selected. You will often see lots of these computed values here, this just means Terraform does not know what these values are yet, but they will be computed eventually after our run. For example, we do not know what the instance ID will be, before the host is actually created, so this makes sense. Then down here, we get a summary of what is going to happen. This is a pretty simple change, but imagine you wanted to deploy a big architecture, so this is a nice sanity check.
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ aws_instance.web1
id:
ami: "ami-032509850cf9ee54e"
arn:
associate_public_ip_address:
availability_zone:
cpu_core_count:
cpu_threads_per_core:
ebs_block_device.#:
ephemeral_block_device.#:
get_password_data: "false"
host_id:
instance_state:
instance_type: "t1.micro"
ipv6_address_count:
ipv6_addresses.#:
key_name:
network_interface.#:
network_interface_id:
password_data:
placement_group:
primary_network_interface_id:
private_dns:
private_ip:
public_dns:
public_ip:
root_block_device.#:
security_groups.#:
source_dest_check: "true"
subnet_id:
tenancy:
volume_tags.%:
vpc_security_group_ids.#:
Plan: 1 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
Now, lets run, terraform apply, to execute our changes. This should look familiar, as this is a generated plan of what is going to happen and we are asked for confirm we still want this. I am going to type, yes, here. Terraform is now connecting to AWS and deploying our infrastructure changes behind the scenes.
Lets change over to the AWS console and see if our instance was created. Just refresh the instance list here. Pretty awesome, we have our instance here and it is still initializing some checks. But, what about making changes to this instance? Lets jump back to the editor for a minute. So, in this block here we have our web1 instance, that we just created, and I now want to tag this instance with the name web1-tf by uncommenting this. By the way, you can find the supported syntax for these blocks on the terraform website, the docs are really good.
$ terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ aws_instance.web1
id:
ami: "ami-032509850cf9ee54e"
arn:
associate_public_ip_address:
availability_zone:
cpu_core_count:
cpu_threads_per_core:
ebs_block_device.#:
ephemeral_block_device.#:
get_password_data: "false"
host_id:
instance_state:
instance_type: "t1.micro"
ipv6_address_count:
ipv6_addresses.#:
key_name:
network_interface.#:
network_interface_id:
password_data:
placement_group:
primary_network_interface_id:
private_dns:
private_ip:
public_dns:
public_ip:
root_block_device.#:
security_groups.#:
source_dest_check: "true"
subnet_id:
tenancy:
volume_tags.%:
vpc_security_group_ids.#:
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_instance.web1: Creating...
ami: "" => "ami-032509850cf9ee54e"
arn: "" => ""
associate_public_ip_address: "" => ""
availability_zone: "" => ""
cpu_core_count: "" => ""
cpu_threads_per_core: "" => ""
ebs_block_device.#: "" => ""
ephemeral_block_device.#: "" => ""
get_password_data: "" => "false"
host_id: "" => ""
instance_state: "" => ""
instance_type: "" => "t1.micro"
ipv6_address_count: "" => ""
ipv6_addresses.#: "" => ""
key_name: "" => ""
network_interface.#: "" => ""
network_interface_id: "" => ""
password_data: "" => ""
placement_group: "" => ""
primary_network_interface_id: "" => ""
private_dns: "" => ""
private_ip: "" => ""
public_dns: "" => ""
public_ip: "" => ""
root_block_device.#: "" => ""
security_groups.#: "" => ""
source_dest_check: "" => "true"
subnet_id: "" => ""
tenancy: "" => ""
volume_tags.%: "" => ""
vpc_security_group_ids.#: "" => ""
aws_instance.web1: Still creating... (10s elapsed)
aws_instance.web1: Creation complete after 16s (ID: i-0934276c941295aa0)
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
I am going to save this and jump over to the command line. Just to recap, we now know what terraform init, plan, and apply mean. But, let me you show you, the show command. This option, basically dumps out the resources terraform is keeping an eye on. It know about our web1 instance here, the instance id, network information, etc.
$ terraform show aws_instance.web1: id = i-0934276c941295aa0 ami = ami-032509850cf9ee54e arn = arn:aws:ec2:us-west-2:590820421865:instance/i-0934276c941295aa0 availability_zone = us-west-2a cpu_core_count = 1 cpu_threads_per_core = 1 credit_specification.# = 1 credit_specification.0.cpu_credits = standard disable_api_termination = false ebs_block_device.# = 0 ebs_optimized = false ephemeral_block_device.# = 0 get_password_data = false iam_instance_profile = instance_state = running instance_type = t1.micro ipv6_addresses.# = 0 key_name = monitoring = false network_interface.# = 0 network_interface_id = password_data = placement_group = primary_network_interface_id = private_dns = ip-10-24-127-204.us-west-2.compute.internal private_ip = 10.24.127.204 public_dns = ec2-35-162-187-244.us-west-2.compute.amazonaws.com public_ip = 35.162.187.244 root_block_device.# = 1 root_block_device.0.delete_on_termination = true root_block_device.0.iops = 100 root_block_device.0.volume_id = vol-0b9531928d58dfa31 root_block_device.0.volume_size = 8 root_block_device.0.volume_type = gp2 security_groups.# = 1 security_groups.3814588639 = default source_dest_check = true subnet_id = tags.% = 0 tenancy = default volume_tags.% = 0 vpc_security_group_ids.# = 0
This data is cached in this state file here that terraform creates. Be a little careful about where you put this state file as it likely contain private details about your infrastructures inner workings.
$ ls -l -rw-r--r-- 1 jweissig staff 440 19 Feb 07:20 main.tf -rw-r--r-- 1 jweissig staff 3718 19 Feb 07:13 terraform.tfstate
To change the web1 instance tag, lets run, terraform plan. We get back an update in-place summary this time, for web1. Saying, hey, I am going to update the tag name for this instance, which makes sense. You will notice that terraform already knows about web1 too, since it was tracking it in that state file, and we are only updating the instance and not recreating it.
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
aws_instance.web1: Refreshing state... (ID: i-0934276c941295aa0)
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
~ aws_instance.web1
tags.%: "0" => "1"
tags.Name: "" => "web1-tf"
Plan: 0 to add, 1 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
You will see a recurring pattern with terraform, in that we pretty much always run, terraform plan, review the output, and if it makes sense, run terraform apply. Okay, the change looks good, lets type in, yes, to make it so. Cool, it says it was updated. Lets check the web console to verify.
$ terraform apply
aws_instance.web1: Refreshing state... (ID: i-0934276c941295aa0)
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
~ aws_instance.web1
tags.%: "0" => "1"
tags.Name: "" => "web1-tf"
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_instance.web1: Modifying... (ID: i-0934276c941295aa0)
tags.%: "0" => "1"
tags.Name: "" => "web1-tf"
aws_instance.web1: Modifications complete after 2s (ID: i-0934276c941295aa0)
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
So, this proves things out from the perspective of creating and updating things. But, what about deleting them? Lets hop back to the command-line and delete the web1 instance. We do this by running, terraform destroy. This refreshes the state, and we see the action here is that it wants to delete something, and it is our web1 instance. Looks okay, so lets delete it. You can see status updates here, but lets check the web console and see if it is updated there too. Yup, our instance is terminated. Back at the command-line, it looks like things were successfully deleted, as we already verified. Easy enough, and hopefully this shows you the power of terraform, albeit in a pretty simple way.
$ terraform destroy aws_instance.web1: Refreshing state... (ID: i-0934276c941295aa0) An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: - aws_instance.web1 Plan: 0 to add, 0 to change, 1 to destroy. Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes aws_instance.web1: Destroying... (ID: i-0934276c941295aa0) aws_instance.web1: Still destroying... (ID: i-0934276c941295aa0, 10s elapsed) aws_instance.web1: Still destroying... (ID: i-0934276c941295aa0, 20s elapsed) aws_instance.web1: Still destroying... (ID: i-0934276c941295aa0, 30s elapsed) aws_instance.web1: Destruction complete after 30s Destroy complete! Resources: 1 destroyed.
So, that is the basics of Terraform. You run terraform init to download supporting packages for what you are attempting to do. Then you run terraform plan, and if all looks look, then terraform apply. You will get into the cycle of planning and applying your infrastructure changes though configuration files. This is where the term, infrastructure as code fits, as you are basically describing infrastructure as code. Very much like configuration management but for infrastructure. Then, eventually you might destroy it.
Sure, this is cool, but why use Terraform? It sort of seems like more upfront work? Well, I wanted to spend a few minutes and just chat about this topic. Basically wanted to chat about a few places and conditions, where using infrastructure automation can save you tons of time and reduce stress.
Take this architecture for example. Maybe this is a mobile games company and they are hosting their backend user database, and store, in the cloud. You have tons of user traffic hitting your clusters, and you have a handful of developers, operations, and data science folks.
What are some reasons you might want to use infrastructure automation here? Maybe you are doing things pretty manually today. I do this sometimes too, I am not trying to force automation on everyone! But, say you are chatting with the CTO and he mentions that traffic is growing, and they plan to hire more developers, and want to start load testing things. The company is finally taking off. He is suggesting creating three identical clusters, one for production, one for dev/test, and another for load testing (this will be spun up on the fly when needed). If you are not using infrastructure automation tools, this is going to be extremely painful, as you are potentially tripling your workload. Even if you manage to do it manually, these clusters will get out of sync really quickly, as configuration drifts between them, and might cause lots of problems. Infrastructure automation tools could help setup, and keep these things in sync, saving you tons of time and stress.
But, how do you start using automation tools like terraform from nothing? Well, I would start small, and build your confidence in the tool. Pick something low rick, where you can start to use automation, and build up your skill and confidence. For example, with this storage system here, maybe you have some buckets that you need to create, along with permissions, and some lifecycle policies. This might be a good start project.
From there, just expand it across your infrastructure, piece by piece. Pretty soon you will have a stockpile of terraform configuration files that describe your infrastructure. Honestly, there is already tons of existing terraform templates out there, that it might not even be that hard. This could also greatly help you, as the team expands, and they start requesting infrastructure changes to support new projects.
Another example, is when things are moving so quickly that you cannot even keep up. The company decides to clone you. You finally get some extra help. With something like Terraform, it allows you to distribute knowledge quickly amongst the team because it is documented. Wiki pages get outdated quickly. Your new team members can self educate. With these Terraform files and a quick whiteboarding session you could transfer a massive amount of knowledge. A big bonus, is that you are now okay to forget stuff, and know that you can understand it again. If you stash these terraform files in something like git, then you can have diffs too, which is pretty awesome.
Lets change gears for a second to another use case. Maybe you are doing some training, maybe helping out at a local meetup, and need to configure a bunch of virtual machines for students. This is a perfect use case for terraform. Or, maybe your job requires you to do lots demos. Again, this could be really useful.
You get the idea now, so I wanted to move on to the second demo. Lets jump back to our editor. So, we already looked at example 1, where we deployed a single web instance. We are going to continue on this theme, in example 2, but expand things a little.
$ ls -l -rw-r--r-- 1 jweissig staff 839 19 Feb 06:05 README.md drwxr-xr-x 6 jweissig staff 192 19 Feb 07:29 example_1 drwxr-xr-x 3 jweissig staff 96 19 Feb 05:42 example_2 -rwxrwxr-x 1 jweissig staff 106576920 14 Dec 13:20 terraform -rw-r--r-- 1 jweissig staff 22431061 19 Feb 06:50 terraform_0.11.11_darwin_amd64.zip $ cd example_2 $ ls -l -rw-r--r-- 1 jweissig staff 2201 19 Feb 13:21 main.tf
Up here we are defining our access keys and region again. Then down here, we have our web1 instance again, but we adding in this user_data field, that will get executed when the instance boots up. Basically, I want to install nginx, and create a default web page. This will allow us to connect to this machine when it boots.
Next, I added this security group line, we are going to use this to define a security group for our web instances, so they they accept external traffic on port 80. Basically, opening up firewall rules here. Actually, let me just show you a diagram of what we are building here to help flush things out. We are creating two web instances, web1 and web2, along with a load balancer that sits in front. So, in the terraform scripts, we just need to define our two web servers, the security group, and the load balancer. Then run it.
So, that line here adds this instance to the security group for allowing http traffic on port 80. This bit here bootstraps the web server instance. A bit hacky but it works. Then down here I am creating a second web server instance. You can actually create both these from within a single block, using something like a counter, but I just did it this way.
Here is the security group for allowing access. Actually, you might notice something interesting here, this is called aws_security_group, and then up in the web server definition, we are referencing this security group variable id. However, this brings up an interesting point, in that we need to security group to be created before the web servers, so that we can have them associated with that security group. I want to show you another example of this.
If we scroll down here, to the load balancer configuration block. The interesting thing here, is where we tell the load balancer what instances to direct traffic to, but these instances do not even exist yet. We are sort of just referencing them in code, but we do not have the real AWS instances IDs, or anything like that yet. So, how does that work? Well, terraform will read in your configuration and build out a dependency graph. For us, it will look something like this. We need to create the security group first, as our web instances need it, then we create the web instances, as the load balancer needs them, and finally we create the load balancer. That is why we are using these references like that.
// configure API access (user with AmazonEC2FullAccess)
provider "aws" {
access_key = "ACCESS_KEY_HERE"
secret_key = "SECRET_KEY_HERE"
region = "us-west-2"
}
// Amazon Linux 2 AMI (HVM), SSD Volume, 64-bit x86
// https://www.terraform.io/docs/providers/aws/r/instance.html
// https://www.terraform.io/docs/providers/aws/r/instance.html#user_data
// https://aws.amazon.com/amazon-linux-2/faqs/#Amazon_Linux_Extras
resource "aws_instance" "web1" {
ami = "ami-032509850cf9ee54e"
instance_type = "t1.micro"
vpc_security_group_ids = ["${aws_security_group.web.id}"]
user_data = <<-EOF
#!/bin/bash
yum update -y
amazon-linux-extras install nginx1.12
echo "#57 - Terraform (web1)" > /usr/share/nginx/html/index.html
systemctl enable nginx
systemctl start nginx
EOF
tags {
Name = "web1-tf"
}
}
// web2
resource "aws_instance" "web2" {
ami = "ami-032509850cf9ee54e"
instance_type = "t1.micro"
vpc_security_group_ids = ["${aws_security_group.web.id}"]
user_data = <<-EOF
#!/bin/bash
yum update -y
amazon-linux-extras install nginx1.12
echo "#57 - Terraform (web2)" > /usr/share/nginx/html/index.html
systemctl enable nginx
systemctl start nginx
EOF
tags {
Name = "web2-tf"
}
}
// https://www.terraform.io/docs/providers/aws/d/security_group.html
resource "aws_security_group" "web" {
name = "ingress-web-http"
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
}
// https://www.terraform.io/docs/providers/aws/r/elb.html
resource "aws_elb" "tf-elb" {
name = "tf-elb"
availability_zones = ["us-west-2a", "us-west-2b", "us-west-2c"]
listener {
instance_port = 80
instance_protocol = "http"
lb_port = 80
lb_protocol = "http"
}
health_check {
healthy_threshold = 2
unhealthy_threshold = 2
timeout = 3
target = "HTTP:80/"
interval = 30
}
instances = ["${aws_instance.web1.id}", "${aws_instance.web2.id}"]
tags = {
Name = "tf-elb"
}
}
Lets jump back to the command-line and walk through second example. I am going to change into the example 2 directory, where we have our terraform configuration file. By the way, in another window, I updated the access and secret token keys, to allow access into my AWS account.
Lets run, terraform init, to download the aws provider dependency again. It just takes a second.
$ terraform init Initializing provider plugins... - Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "aws" (1.59.0)... The following providers do not have any version constraints in configuration, so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below. * provider.aws: version = "~> 1.59" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
Now, lets run, terraform plan, to generate a report what it thinks will happen, and then lets walk through it. So, we are going to create a bunch of stuff, our load balancer, and again you notice these computed values here, in that it does not know these values until it runs, a good example is our web server instance IDs.
Next, we are going to create web1, using our custom user_data script to bootstrap the web server. Again, we will do the same things for web2. Finally, we are going to create our security group. Looks like everything we expected to happen.
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ aws_elb.tf-elb
id:
arn:
availability_zones.#: "3"
availability_zones.2050015877: "us-west-2c"
availability_zones.221770259: "us-west-2b"
availability_zones.2487133097: "us-west-2a"
connection_draining: "false"
connection_draining_timeout: "300"
cross_zone_load_balancing: "true"
dns_name:
health_check.#: "1"
health_check.0.healthy_threshold: "2"
health_check.0.interval: "30"
health_check.0.target: "HTTP:80/"
health_check.0.timeout: "3"
health_check.0.unhealthy_threshold: "2"
idle_timeout: "60"
instances.#:
internal:
listener.#: "1"
listener.3057123346.instance_port: "80"
listener.3057123346.instance_protocol: "http"
listener.3057123346.lb_port: "80"
listener.3057123346.lb_protocol: "http"
listener.3057123346.ssl_certificate_id: ""
name: "tf-elb"
security_groups.#:
source_security_group:
source_security_group_id:
subnets.#:
tags.%: "1"
tags.Name: "tf-elb"
zone_id:
+ aws_instance.web1
id:
ami: "ami-032509850cf9ee54e"
arn:
associate_public_ip_address:
availability_zone:
cpu_core_count:
cpu_threads_per_core:
ebs_block_device.#:
ephemeral_block_device.#:
get_password_data: "false"
host_id:
instance_state:
instance_type: "t1.micro"
ipv6_address_count:
ipv6_addresses.#:
key_name:
network_interface.#:
network_interface_id:
password_data:
placement_group:
primary_network_interface_id:
private_dns:
private_ip:
public_dns:
public_ip:
root_block_device.#:
security_groups.#:
source_dest_check: "true"
subnet_id:
tags.%: "1"
tags.Name: "web1-tf"
tenancy:
user_data: "c3c505ea292279ddd315bdb3ec64e05192e19d3f"
volume_tags.%:
vpc_security_group_ids.#:
+ aws_instance.web2
id:
ami: "ami-032509850cf9ee54e"
arn:
associate_public_ip_address:
availability_zone:
cpu_core_count:
cpu_threads_per_core:
ebs_block_device.#:
ephemeral_block_device.#:
get_password_data: "false"
host_id:
instance_state:
instance_type: "t1.micro"
ipv6_address_count:
ipv6_addresses.#:
key_name:
network_interface.#:
network_interface_id:
password_data:
placement_group:
primary_network_interface_id:
private_dns:
private_ip:
public_dns:
public_ip:
root_block_device.#:
security_groups.#:
source_dest_check: "true"
subnet_id:
tags.%: "1"
tags.Name: "web2-tf"
tenancy:
user_data: "4a1d5424417563116508171d1daa988ab6eb607a"
volume_tags.%:
vpc_security_group_ids.#:
+ aws_security_group.web
id:
arn:
description: "Managed by Terraform"
egress.#:
ingress.#: "1"
ingress.2214680975.cidr_blocks.#: "1"
ingress.2214680975.cidr_blocks.0: "0.0.0.0/0"
ingress.2214680975.description: ""
ingress.2214680975.from_port: "80"
ingress.2214680975.ipv6_cidr_blocks.#: "0"
ingress.2214680975.prefix_list_ids.#: "0"
ingress.2214680975.protocol: "tcp"
ingress.2214680975.security_groups.#: "0"
ingress.2214680975.self: "false"
ingress.2214680975.to_port: "80"
name: "ingress-web-http"
owner_id:
revoke_rules_on_delete: "false"
vpc_id:
Plan: 4 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
Lets run, terraform apply, and make create this infrastructure. Again, we are asked for confirm this, by typing yes here, and away it goes. Lets jump over to the AWS web interface and see if we can catch some of this stuff being created. Great, we can see our two web instances being created here.
Lets scrolls down and see if we can catch the security group. Yup, it is here too. Lets verify it is allowing ingress on port 80. Perfect. Finally, lets check out the load balancer. Great, we have our load balancer up and running now too. Lets just check on our instances again and see where they are at in the booting process. Great, looks like it is all up and running, this is pretty cool.
$ terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ aws_elb.tf-elb
id:
arn:
availability_zones.#: "3"
availability_zones.2050015877: "us-west-2c"
availability_zones.221770259: "us-west-2b"
availability_zones.2487133097: "us-west-2a"
connection_draining: "false"
connection_draining_timeout: "300"
cross_zone_load_balancing: "true"
dns_name:
health_check.#: "1"
health_check.0.healthy_threshold: "2"
health_check.0.interval: "30"
health_check.0.target: "HTTP:80/"
health_check.0.timeout: "3"
health_check.0.unhealthy_threshold: "2"
idle_timeout: "60"
instances.#:
internal:
listener.#: "1"
listener.3057123346.instance_port: "80"
listener.3057123346.instance_protocol: "http"
listener.3057123346.lb_port: "80"
listener.3057123346.lb_protocol: "http"
listener.3057123346.ssl_certificate_id: ""
name: "tf-elb"
security_groups.#:
source_security_group:
source_security_group_id:
subnets.#:
tags.%: "1"
tags.Name: "tf-elb"
zone_id:
+ aws_instance.web1
id:
ami: "ami-032509850cf9ee54e"
arn:
associate_public_ip_address:
availability_zone:
cpu_core_count:
cpu_threads_per_core:
ebs_block_device.#:
ephemeral_block_device.#:
get_password_data: "false"
host_id:
instance_state:
instance_type: "t1.micro"
ipv6_address_count:
ipv6_addresses.#:
key_name:
network_interface.#:
network_interface_id:
password_data:
placement_group:
primary_network_interface_id:
private_dns:
private_ip:
public_dns:
public_ip:
root_block_device.#:
security_groups.#:
source_dest_check: "true"
subnet_id:
tags.%: "1"
tags.Name: "web1-tf"
tenancy:
user_data: "c3c505ea292279ddd315bdb3ec64e05192e19d3f"
volume_tags.%:
vpc_security_group_ids.#:
+ aws_instance.web2
id:
ami: "ami-032509850cf9ee54e"
arn:
associate_public_ip_address:
availability_zone:
cpu_core_count:
cpu_threads_per_core:
ebs_block_device.#:
ephemeral_block_device.#:
get_password_data: "false"
host_id:
instance_state:
instance_type: "t1.micro"
ipv6_address_count:
ipv6_addresses.#:
key_name:
network_interface.#:
network_interface_id:
password_data:
placement_group:
primary_network_interface_id:
private_dns:
private_ip:
public_dns:
public_ip:
root_block_device.#:
security_groups.#:
source_dest_check: "true"
subnet_id:
tags.%: "1"
tags.Name: "web2-tf"
tenancy:
user_data: "4a1d5424417563116508171d1daa988ab6eb607a"
volume_tags.%:
vpc_security_group_ids.#:
+ aws_security_group.web
id:
arn:
description: "Managed by Terraform"
egress.#:
ingress.#: "1"
ingress.2214680975.cidr_blocks.#: "1"
ingress.2214680975.cidr_blocks.0: "0.0.0.0/0"
ingress.2214680975.description: ""
ingress.2214680975.from_port: "80"
ingress.2214680975.ipv6_cidr_blocks.#: "0"
ingress.2214680975.prefix_list_ids.#: "0"
ingress.2214680975.protocol: "tcp"
ingress.2214680975.security_groups.#: "0"
ingress.2214680975.self: "false"
ingress.2214680975.to_port: "80"
name: "ingress-web-http"
owner_id:
revoke_rules_on_delete: "false"
vpc_id:
Plan: 4 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_security_group.web: Creating...
arn: "" => ""
description: "" => "Managed by Terraform"
egress.#: "" => ""
ingress.#: "" => "1"
ingress.2214680975.cidr_blocks.#: "" => "1"
ingress.2214680975.cidr_blocks.0: "" => "0.0.0.0/0"
ingress.2214680975.description: "" => ""
ingress.2214680975.from_port: "" => "80"
ingress.2214680975.ipv6_cidr_blocks.#: "" => "0"
ingress.2214680975.prefix_list_ids.#: "" => "0"
ingress.2214680975.protocol: "" => "tcp"
ingress.2214680975.security_groups.#: "" => "0"
ingress.2214680975.self: "" => "false"
ingress.2214680975.to_port: "" => "80"
name: "" => "ingress-web-http"
owner_id: "" => ""
revoke_rules_on_delete: "" => "false"
vpc_id: "" => ""
aws_security_group.web: Creation complete after 1s (ID: sg-0f3f9203cc504f28d)
aws_instance.web1: Creating...
ami: "" => "ami-032509850cf9ee54e"
arn: "" => ""
associate_public_ip_address: "" => ""
availability_zone: "" => ""
cpu_core_count: "" => ""
cpu_threads_per_core: "" => ""
ebs_block_device.#: "" => ""
ephemeral_block_device.#: "" => ""
get_password_data: "" => "false"
host_id: "" => ""
instance_state: "" => ""
instance_type: "" => "t1.micro"
ipv6_address_count: "" => ""
ipv6_addresses.#: "" => ""
key_name: "" => ""
network_interface.#: "" => ""
network_interface_id: "" => ""
password_data: "" => ""
placement_group: "" => ""
primary_network_interface_id: "" => ""
private_dns: "" => ""
private_ip: "" => ""
public_dns: "" => ""
public_ip: "" => ""
root_block_device.#: "" => ""
security_groups.#: "" => ""
source_dest_check: "" => "true"
subnet_id: "" => ""
tags.%: "" => "1"
tags.Name: "" => "web1-tf"
tenancy: "" => ""
user_data: "" => "c3c505ea292279ddd315bdb3ec64e05192e19d3f"
volume_tags.%: "" => ""
vpc_security_group_ids.#: "" => "1"
vpc_security_group_ids.1019125703: "" => "sg-0f3f9203cc504f28d"
aws_instance.web2: Creating...
ami: "" => "ami-032509850cf9ee54e"
arn: "" => ""
associate_public_ip_address: "" => ""
availability_zone: "" => ""
cpu_core_count: "" => ""
cpu_threads_per_core: "" => ""
ebs_block_device.#: "" => ""
ephemeral_block_device.#: "" => ""
get_password_data: "" => "false"
host_id: "" => ""
instance_state: "" => ""
instance_type: "" => "t1.micro"
ipv6_address_count: "" => ""
ipv6_addresses.#: "" => ""
key_name: "" => ""
network_interface.#: "" => ""
network_interface_id: "" => ""
password_data: "" => ""
placement_group: "" => ""
primary_network_interface_id: "" => ""
private_dns: "" => ""
private_ip: "" => ""
public_dns: "" => ""
public_ip: "" => ""
root_block_device.#: "" => ""
security_groups.#: "" => ""
source_dest_check: "" => "true"
subnet_id: "" => ""
tags.%: "" => "1"
tags.Name: "" => "web2-tf"
tenancy: "" => ""
user_data: "" => "4a1d5424417563116508171d1daa988ab6eb607a"
volume_tags.%: "" => ""
vpc_security_group_ids.#: "" => "1"
vpc_security_group_ids.1019125703: "" => "sg-0f3f9203cc504f28d"
aws_instance.web2: Still creating... (10s elapsed)
aws_instance.web1: Still creating... (10s elapsed)
aws_instance.web1: Still creating... (20s elapsed)
aws_instance.web2: Still creating... (20s elapsed)
aws_instance.web2: Creation complete after 22s (ID: i-07cfbc721c4f84e23)
aws_instance.web1: Still creating... (30s elapsed)
aws_instance.web1: Creation complete after 33s (ID: i-05fe82cc429a277f4)
aws_elb.tf-elb: Creating...
arn: "" => ""
availability_zones.#: "" => "3"
availability_zones.2050015877: "" => "us-west-2c"
availability_zones.221770259: "" => "us-west-2b"
availability_zones.2487133097: "" => "us-west-2a"
connection_draining: "" => "false"
connection_draining_timeout: "" => "300"
cross_zone_load_balancing: "" => "true"
dns_name: "" => ""
health_check.#: "" => "1"
health_check.0.healthy_threshold: "" => "2"
health_check.0.interval: "" => "30"
health_check.0.target: "" => "HTTP:80/"
health_check.0.timeout: "" => "3"
health_check.0.unhealthy_threshold: "" => "2"
idle_timeout: "" => "60"
instances.#: "" => "2"
instances.2195798590: "" => "i-05fe82cc429a277f4"
instances.2556045284: "" => "i-07cfbc721c4f84e23"
internal: "" => ""
listener.#: "" => "1"
listener.3057123346.instance_port: "" => "80"
listener.3057123346.instance_protocol: "" => "http"
listener.3057123346.lb_port: "" => "80"
listener.3057123346.lb_protocol: "" => "http"
listener.3057123346.ssl_certificate_id: "" => ""
name: "" => "tf-elb"
security_groups.#: "" => ""
source_security_group: "" => ""
source_security_group_id: "" => ""
subnets.#: "" => ""
tags.%: "" => "1"
tags.Name: "" => "tf-elb"
zone_id: "" => ""
aws_elb.tf-elb: Creation complete after 3s (ID: tf-elb)
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
Lets jump back to the command-line and see what terraform says. Great, it is reporting that things were successful too. If you scroll up, there are lots of messages about how things are processing, which is pretty nice especially when it takes a while.
Lets jump back to the web console and try connecting to our two web servers. If you click on the instance name here, we can get the external IP address, lets copy that and try connecting to it. Awesome, so we have web1 up and running! This means, at this point, we can verify that terraform correctly booted web1, pushed the user_data script and installed the web server, and our security group is working correctly since we can connect to this instance.
Lets try web2 now. Again, we will just copy the external IP address and connect to it. Perfect, so we have both web1 and web2 working as expected. This is pretty cool. But, lets check out the load balancer too, as that will sit in front and balancer traffic between these two instances. I am just going to go down to the load balancer, we will copy its frontend hostname, and try connecting to it. Sweet, it works, so we have a complete system now, two web instances with a load balancer sitting in front. If I refresh the page here you can see it splitting traffic between web1 and web2.
Actually, let me just make this bigger, so it is easier to see. Now lets refresh the page a few times. Pretty cool, right? So, this is what we built out. Even though this is pretty simple, hopefully this gives you a taste of what terraform is, and how it works. My personal suggestion would be to just start really simple, build confidence, and then see what might help you out in your workflow.
We should clean all this up before we leave. Lets jump back to the command-line and run terraform destroy. This will inspect our current state, then ask if we are sure we want to blow this all away, and yes we do. I just wanted to check out the console quickly and see if we can catch any of this being deleted. Yup, looks like the instances are going down, security group is still here, load balancer is deleted, and the security group is gone now too. Pretty cool. This might also be a little scary, in that with a single command, you can blow away a huge amount of infrastructure, so that is what I was sort of suggesting to start slow and build you confidence in what this tool can do. The worst feeling in doing a deploy and then getting a bunch of monitoring alerts that things just broke!
$ terraform destroy aws_security_group.web: Refreshing state... (ID: sg-0f3f9203cc504f28d) aws_instance.web2: Refreshing state... (ID: i-07cfbc721c4f84e23) aws_instance.web1: Refreshing state... (ID: i-05fe82cc429a277f4) aws_elb.tf-elb: Refreshing state... (ID: tf-elb) An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: - aws_elb.tf-elb - aws_instance.web1 - aws_instance.web2 - aws_security_group.web Plan: 0 to add, 0 to change, 4 to destroy. Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes aws_elb.tf-elb: Destroying... (ID: tf-elb) aws_elb.tf-elb: Destruction complete after 0s aws_instance.web2: Destroying... (ID: i-07cfbc721c4f84e23) aws_instance.web1: Destroying... (ID: i-05fe82cc429a277f4) aws_instance.web2: Still destroying... (ID: i-07cfbc721c4f84e23, 10s elapsed) aws_instance.web1: Still destroying... (ID: i-05fe82cc429a277f4, 10s elapsed) aws_instance.web2: Still destroying... (ID: i-07cfbc721c4f84e23, 20s elapsed) aws_instance.web1: Still destroying... (ID: i-05fe82cc429a277f4, 20s elapsed) aws_instance.web2: Still destroying... (ID: i-07cfbc721c4f84e23, 30s elapsed) aws_instance.web1: Still destroying... (ID: i-05fe82cc429a277f4, 30s elapsed) aws_instance.web2: Destruction complete after 31s aws_instance.web1: Destruction complete after 31s aws_security_group.web: Destroying... (ID: sg-0f3f9203cc504f28d) aws_security_group.web: Destruction complete after 0s Destroy complete! Resources: 4 destroyed.
Before we end this episode, I wanted to show you a couple links to check out if you are interested in learning more. The documentation on terraform is pretty extensive. Terraform supports all types of things, like AWS, Google Cloud, Azure, Cloudflare, VMware, etc. It is pretty crazy. Lots of these are community contributed and you can really leverage lots of things without reinventing the wheel. If you click through here there is tons of examples and documentation too.
This Terraform learning page is also really awesome. It walks you through lots of the fundamentals in depth. Which is pretty nice. Finally, there is a site called the Terraform Module Registry, where you can find tons of examples. This is where AWS has something like 700 plus modules. All of these links are in the episode notes below. I also posted my examples to github if you wanted to have a look. Honestly, I would just use the ones on the terraform site though.
Alright, that’s it for this episode. Thanks for watching. I’ll see you next week. Bye.