Terraform use S3 as remote backend & DynamoDB to track the lock
NOTE: All configurations were taken from a lab environment.
Introduction
This guide will show how to use S3 as backend and DynamoDB to control the lock in Terraform.
When Terraform runs, it creates a Terraform state file to track the infrastructure it has created. This file is named terraform.tfstate and it is created in the same folder where Terraform ran.
It is very important to keep this file in order to track the infrastructure changes done with Terraform. Everytime we run Terraform in the folder, the diffs are checked agains the terraform.tfstate file. This file should never be edited by hand.
Create S3 & DynamoDB table with Terraform
Using S3 as remote backend, Terraform will load and update the file in S3 every time the plan or apply command ran. Using DynamoDB will record a lock usage. This so only a single user can run Terraform at the time. The lock is acquired when running apply and released once it completes.
Use apply -lock-timeout=<TIME> in order to set the duration to retry a state lock
To wait 10 seconds use:
1
terraform apply -lock-timeout=10s
To wait 10 minutes use:
1
terraform apply -lock-timeout=10m
We can use Terraform to create the S3 Bucket and DynamoDB Table
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
provider "aws" {
region = "us-east-1"
}
resource "aws_s3_bucket" "terraform_state" {
bucket = "terraform-backend-20240724212654"
# Prevent accidental deletion of this S3 bucket
lifecycle {
prevent_destroy = true
}
}
# Enable versioning to see the full revision history of state files
resource "aws_s3_bucket_versioning" "enabled" {
bucket = aws_s3_bucket.terraform_state.id
versioning_configuration {
status = "Enabled"
}
}
# Enable server-side encryption by default
resource "aws_s3_bucket_server_side_encryption_configuration" "default" {
bucket = aws_s3_bucket.terraform_state.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
# Explicitly block all public access to the S3 bucket
resource "aws_s3_bucket_public_access_block" "public_access" {
bucket = aws_s3_bucket.terraform_state.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
resource "aws_dynamodb_table" "terraform_locks" {
name = "terraform-up-and-running-locks"
billing_mode = "PAY_PER_REQUEST"
hash_key = "LockID"
attribute {
name = "LockID"
type = "S"
}
}
Resource Documentation:
- Resource: aws_s3_bucket
- Resource: aws_s3_bucket_versioning
- Resource: aws_s3_bucket_server_side_encryption_configuration
- Resource: aws_s3_bucket_public_access_block
- Resource: aws_dynamodb_table
terraform init
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
cloud_user@553b1e446c1c:~$ mkdir terraform_backend
cloud_user@553b1e446c1c:~$ cd terraform_backend/
cloud_user@553b1e446c1c:~/terraform_backend$ vim main.tf
cloud_user@553b1e446c1c:~/terraform_backend$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v5.59.0...
- Installed hashicorp/aws v5.59.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
cloud_user@553b1e446c1c:~/terraform_backend$
terraform plan
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
cloud_user@553b1e446c1c:~/terraform_backend$ terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_dynamodb_table.terraform_locks will be created
+ resource "aws_dynamodb_table" "terraform_locks" {
+ arn = (known after apply)
+ billing_mode = "PAY_PER_REQUEST"
+ hash_key = "LockID"
+ id = (known after apply)
+ name = "terraform-up-and-running-locks"
+ read_capacity = (known after apply)
+ stream_arn = (known after apply)
+ stream_label = (known after apply)
+ stream_view_type = (known after apply)
+ tags_all = (known after apply)
+ write_capacity = (known after apply)
+ attribute {
+ name = "LockID"
+ type = "S"
}
+ point_in_time_recovery (known after apply)
+ server_side_encryption (known after apply)
+ ttl (known after apply)
}
# aws_s3_bucket.terraform_state will be created
+ resource "aws_s3_bucket" "terraform_state" {
+ acceleration_status = (known after apply)
+ acl = (known after apply)
+ arn = (known after apply)
+ bucket = "terraform-backend-20240724212654"
+ bucket_domain_name = (known after apply)
+ bucket_prefix = (known after apply)
+ bucket_regional_domain_name = (known after apply)
+ force_destroy = false
+ hosted_zone_id = (known after apply)
+ id = (known after apply)
+ object_lock_enabled = (known after apply)
+ policy = (known after apply)
+ region = (known after apply)
+ request_payer = (known after apply)
+ tags_all = (known after apply)
+ website_domain = (known after apply)
+ website_endpoint = (known after apply)
+ cors_rule (known after apply)
+ grant (known after apply)
+ lifecycle_rule (known after apply)
+ logging (known after apply)
+ object_lock_configuration (known after apply)
+ replication_configuration (known after apply)
+ server_side_encryption_configuration (known after apply)
+ versioning (known after apply)
+ website (known after apply)
}
# aws_s3_bucket_public_access_block.public_access will be created
+ resource "aws_s3_bucket_public_access_block" "public_access" {
+ block_public_acls = true
+ block_public_policy = true
+ bucket = (known after apply)
+ id = (known after apply)
+ ignore_public_acls = true
+ restrict_public_buckets = true
}
# aws_s3_bucket_server_side_encryption_configuration.default will be created
+ resource "aws_s3_bucket_server_side_encryption_configuration" "default" {
+ bucket = (known after apply)
+ id = (known after apply)
+ rule {
+ apply_server_side_encryption_by_default {
+ sse_algorithm = "AES256"
# (1 unchanged attribute hidden)
}
}
}
# aws_s3_bucket_versioning.enabled will be created
+ resource "aws_s3_bucket_versioning" "enabled" {
+ bucket = (known after apply)
+ id = (known after apply)
+ versioning_configuration {
+ mfa_delete = (known after apply)
+ status = "Enabled"
}
}
Plan: 5 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
cloud_user@553b1e446c1c:~/terraform_backend$
terraform apply
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
cloud_user@553b1e446c1c:~/terraform_backend$ terraform apply
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_dynamodb_table.terraform_locks will be created
+ resource "aws_dynamodb_table" "terraform_locks" {
+ arn = (known after apply)
+ billing_mode = "PAY_PER_REQUEST"
+ hash_key = "LockID"
+ id = (known after apply)
+ name = "terraform-up-and-running-locks"
+ read_capacity = (known after apply)
+ stream_arn = (known after apply)
+ stream_label = (known after apply)
+ stream_view_type = (known after apply)
+ tags_all = (known after apply)
+ write_capacity = (known after apply)
+ attribute {
+ name = "LockID"
+ type = "S"
}
+ point_in_time_recovery (known after apply)
+ server_side_encryption (known after apply)
+ ttl (known after apply)
}
# aws_s3_bucket.terraform_state will be created
+ resource "aws_s3_bucket" "terraform_state" {
+ acceleration_status = (known after apply)
+ acl = (known after apply)
+ arn = (known after apply)
+ bucket = "terraform-backend-20240724212654"
+ bucket_domain_name = (known after apply)
+ bucket_prefix = (known after apply)
+ bucket_regional_domain_name = (known after apply)
+ force_destroy = false
+ hosted_zone_id = (known after apply)
+ id = (known after apply)
+ object_lock_enabled = (known after apply)
+ policy = (known after apply)
+ region = (known after apply)
+ request_payer = (known after apply)
+ tags_all = (known after apply)
+ website_domain = (known after apply)
+ website_endpoint = (known after apply)
+ cors_rule (known after apply)
+ grant (known after apply)
+ lifecycle_rule (known after apply)
+ logging (known after apply)
+ object_lock_configuration (known after apply)
+ replication_configuration (known after apply)
+ server_side_encryption_configuration (known after apply)
+ versioning (known after apply)
+ website (known after apply)
}
# aws_s3_bucket_public_access_block.public_access will be created
+ resource "aws_s3_bucket_public_access_block" "public_access" {
+ block_public_acls = true
+ block_public_policy = true
+ bucket = (known after apply)
+ id = (known after apply)
+ ignore_public_acls = true
+ restrict_public_buckets = true
}
# aws_s3_bucket_server_side_encryption_configuration.default will be created
+ resource "aws_s3_bucket_server_side_encryption_configuration" "default" {
+ bucket = (known after apply)
+ id = (known after apply)
+ rule {
+ apply_server_side_encryption_by_default {
+ sse_algorithm = "AES256"
# (1 unchanged attribute hidden)
}
}
}
# aws_s3_bucket_versioning.enabled will be created
+ resource "aws_s3_bucket_versioning" "enabled" {
+ bucket = (known after apply)
+ id = (known after apply)
+ versioning_configuration {
+ mfa_delete = (known after apply)
+ status = "Enabled"
}
}
Plan: 5 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_dynamodb_table.terraform_locks: Creating...
aws_s3_bucket.terraform_state: Creating...
aws_s3_bucket.terraform_state: Creation complete after 0s [id=terraform-backend-20240724212654]
aws_s3_bucket_versioning.enabled: Creating...
aws_s3_bucket_server_side_encryption_configuration.default: Creating...
aws_s3_bucket_public_access_block.public_access: Creating...
aws_s3_bucket_public_access_block.public_access: Creation complete after 1s [id=terraform-backend-20240724212654]
aws_s3_bucket_server_side_encryption_configuration.default: Creation complete after 1s [id=terraform-backend-20240724212654]
aws_s3_bucket_versioning.enabled: Creation complete after 2s [id=terraform-backend-20240724212654]
aws_dynamodb_table.terraform_locks: Creation complete after 6s [id=terraform-up-and-running-locks]
Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
cloud_user@553b1e446c1c:~/terraform_backend$
The S3 bucked was created
The DynamoDB table was created
Configure Terraform Backend
The syntax to configure a backend is:
1
2
3
4
5
terraform {
backend "<BACKEND_NAME>" {
[CONFIG...]
}
}
Now, we will need to be on a different directory since we don’t want to mix this with the Terraform state to create the S3 bucket and DynamoDB Tables themselves.
1
2
3
cd
mkdir new_terraform_project
cd new_terraform_project
Create a file called backend.tf
1
2
3
4
5
6
7
8
9
10
11
12
terraform {
backend "s3" {
# Replace this with your bucket name!
bucket = "terraform-backend-20240724212654"
key = "global/s3/terraform.tfstate"
region = "us-east-1"
# Replace this with your DynamoDB table name!
dynamodb_table = "terraform-up-and-running-locks"
encrypt = true
}
}
And create main.tf to deploy an EC2 instance for testing purposes.
1
2
3
4
5
6
7
8
9
10
11
12
provider "aws" {
region = "us-east-1"
}
resource "aws_instance" "example" {
ami = "ami-04a81a99f5ec58529"
instance_type = "t2.micro"
tags = {
Name = "ubuntu1-named-with-terraform"
}
}
S3 Backend documentation:
terraform init
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
cloud_user@553b1e446c1c:~/new_terraform_project$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v5.59.0...
- Installed hashicorp/aws v5.59.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
cloud_user@553b1e446c1c:~/new_terraform_project$
terraform plan
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
cloud_user@553b1e446c1c:~/new_terraform_project$ terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_instance.example will be created
+ resource "aws_instance" "example" {
+ ami = "ami-04a81a99f5ec58529"
+ arn = (known after apply)
+ associate_public_ip_address = (known after apply)
+ availability_zone = (known after apply)
+ cpu_core_count = (known after apply)
+ cpu_threads_per_core = (known after apply)
+ disable_api_stop = (known after apply)
+ disable_api_termination = (known after apply)
+ ebs_optimized = (known after apply)
+ get_password_data = false
+ host_id = (known after apply)
+ host_resource_group_arn = (known after apply)
+ iam_instance_profile = (known after apply)
+ id = (known after apply)
+ instance_initiated_shutdown_behavior = (known after apply)
+ instance_lifecycle = (known after apply)
+ instance_state = (known after apply)
+ instance_type = "t2.micro"
+ ipv6_address_count = (known after apply)
+ ipv6_addresses = (known after apply)
+ key_name = (known after apply)
+ monitoring = (known after apply)
+ outpost_arn = (known after apply)
+ password_data = (known after apply)
+ placement_group = (known after apply)
+ placement_partition_number = (known after apply)
+ primary_network_interface_id = (known after apply)
+ private_dns = (known after apply)
+ private_ip = (known after apply)
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ secondary_private_ips = (known after apply)
+ security_groups = (known after apply)
+ source_dest_check = true
+ spot_instance_request_id = (known after apply)
+ subnet_id = (known after apply)
+ tags = {
+ "Name" = "ubuntu1-named-with-terraform"
}
+ tags_all = {
+ "Name" = "ubuntu1-named-with-terraform"
}
+ tenancy = (known after apply)
+ user_data = (known after apply)
+ user_data_base64 = (known after apply)
+ user_data_replace_on_change = false
+ vpc_security_group_ids = (known after apply)
+ capacity_reservation_specification (known after apply)
+ cpu_options (known after apply)
+ ebs_block_device (known after apply)
+ enclave_options (known after apply)
+ ephemeral_block_device (known after apply)
+ instance_market_options (known after apply)
+ maintenance_options (known after apply)
+ metadata_options (known after apply)
+ network_interface (known after apply)
+ private_dns_name_options (known after apply)
+ root_block_device (known after apply)
}
Plan: 1 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
cloud_user@553b1e446c1c:~/new_terraform_project$
terraform apply
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
cloud_user@553b1e446c1c:~/new_terraform_project$ terraform apply
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_instance.example will be created
+ resource "aws_instance" "example" {
+ ami = "ami-04a81a99f5ec58529"
+ arn = (known after apply)
+ associate_public_ip_address = (known after apply)
+ availability_zone = (known after apply)
+ cpu_core_count = (known after apply)
+ cpu_threads_per_core = (known after apply)
+ disable_api_stop = (known after apply)
+ disable_api_termination = (known after apply)
+ ebs_optimized = (known after apply)
+ get_password_data = false
+ host_id = (known after apply)
+ host_resource_group_arn = (known after apply)
+ iam_instance_profile = (known after apply)
+ id = (known after apply)
+ instance_initiated_shutdown_behavior = (known after apply)
+ instance_lifecycle = (known after apply)
+ instance_state = (known after apply)
+ instance_type = "t2.micro"
+ ipv6_address_count = (known after apply)
+ ipv6_addresses = (known after apply)
+ key_name = (known after apply)
+ monitoring = (known after apply)
+ outpost_arn = (known after apply)
+ password_data = (known after apply)
+ placement_group = (known after apply)
+ placement_partition_number = (known after apply)
+ primary_network_interface_id = (known after apply)
+ private_dns = (known after apply)
+ private_ip = (known after apply)
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ secondary_private_ips = (known after apply)
+ security_groups = (known after apply)
+ source_dest_check = true
+ spot_instance_request_id = (known after apply)
+ subnet_id = (known after apply)
+ tags = {
+ "Name" = "ubuntu1-named-with-terraform"
}
+ tags_all = {
+ "Name" = "ubuntu1-named-with-terraform"
}
+ tenancy = (known after apply)
+ user_data = (known after apply)
+ user_data_base64 = (known after apply)
+ user_data_replace_on_change = false
+ vpc_security_group_ids = (known after apply)
+ capacity_reservation_specification (known after apply)
+ cpu_options (known after apply)
+ ebs_block_device (known after apply)
+ enclave_options (known after apply)
+ ephemeral_block_device (known after apply)
+ instance_market_options (known after apply)
+ maintenance_options (known after apply)
+ metadata_options (known after apply)
+ network_interface (known after apply)
+ private_dns_name_options (known after apply)
+ root_block_device (known after apply)
}
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_instance.example: Creating...
aws_instance.example: Still creating... [10s elapsed]
aws_instance.example: Still creating... [20s elapsed]
aws_instance.example: Still creating... [30s elapsed]
aws_instance.example: Creation complete after 32s [id=i-07d7189d66be396cb]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
cloud_user@553b1e446c1c:~/new_terraform_project$
Only after the first terraform apply the terraform.tfstate was created in the S3 bucker
References
This post is licensed under CC BY 4.0 by the author.