Building Secure AWS S3 Infrastructure with Terraform - Complete Lab Guide

S3 buckets are deceptively simple to create and dangerously easy to misconfigure. In this lab, I built a secure S3 infrastructure using Terraform that goes beyond the basics: encrypted storage, least-privilege IAM roles, VPC endpoints for private connectivity, CloudTrail auditing, and lifecycle policies for cost optimization.

Architecture Overview

Infrastructure Components

The lab implements a multi-tier architecture:

Network Layer: Custom VPC with public and private subnets, NAT Gateway for outbound internet access
Compute Layer: Bastion host for secure access, private EC2 instance with IAM role-based S3 access
Storage Layer: Encrypted S3 bucket with versioning, lifecycle policies, and access logging
Security Layer: IAM roles with least-privilege policies, VPC endpoints for private connectivity
Monitoring Layer: CloudTrail for API auditing, S3 access logs for request-level monitoring

Network Architecture

Internet Gateway
       |
   Public Subnet (10.0.1.0/24)
       |
   Bastion Host
       |
   Private Subnet (10.0.2.0/24)
       |
   Private EC2 Instance -----> S3 VPC Endpoint -----> S3 Bucket
       |
   NAT Gateway (for outbound internet access)

┌─────────────────────────────────────────────────────────────┐
│                    My Existing VPC                          │
│  ┌─────────────────┐              ┌─────────────────────┐   │
│  │  Public Subnet  │              │  Private Subnet     │   │
│  │                 │              │                     │   │
│  │  ┌───────────┐  │              │  ┌───────────────┐  │   │
│  │  │ Bastion   │  │              │  │ Private EC2   │  │   │
│  │  │ Host      │  │──────────────│  │ + IAM Role    │  │   │
│  │  │           │  │              │  │               │  │   │
│  │  └───────────┘  │              │  └───────────────┘  │   │
│  └─────────────────┘              └─────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                               │
                               │ IAM Role Permissions
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                    Secure S3 Bucket                         │
│  • Server-Side Encryption (SSE-S3 or SSE-KMS)               │
│  • Public Access Blocked                                    │
│  • Bucket Policy Enforcement                                │
│  • CloudTrail Logging                                       │
└─────────────────────────────────────────────────────────────┘

Infrastructure Implementation

1. VPC and Network Configuration

I started with a custom VPC to provide network isolation and controlled access patterns.

VPC Configuration (vpc.tf)

hljs hcl

resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-vpc"
  })
}

Creates an isolated network with DNS resolution enabled for service discovery within the VPC.

Subnet Architecture

Public Subnet: Hosts the bastion host with direct internet access through an Internet Gateway.

hljs hcl

resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.public_subnet_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-public-subnet"
    Type = "Public"
  })
}

Private Subnet: Hosts application workloads with no direct internet access, using NAT Gateway for outbound connectivity.

hljs hcl

resource "aws_subnet" "private" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.private_subnet_cidr
  availability_zone = data.aws_availability_zones.available.names[0]
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-private-subnet"
    Type = "Private"
  })
}

This defense-in-depth approach isolates internet-facing resources from internal workloads, reducing the attack surface.

NAT Gateway Implementation

hljs hcl

resource "aws_nat_gateway" "main" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public.id
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-nat-gateway"
  })
  
  depends_on = [aws_internet_gateway.main]
}

The NAT Gateway enables outbound internet connectivity for private subnet resources (package updates, AWS API calls) while preventing inbound access.

2. S3 Bucket Configuration

I configured the S3 bucket with multiple layers of protection following AWS security best practices.

Primary S3 Bucket (s3.tf)

hljs hcl

resource "aws_s3_bucket" "secure_bucket" {
  bucket = local.bucket_name
  
  tags = merge(local.common_tags, {
    Name           = "${local.name_prefix}-secure-bucket"
    Purpose        = "Secure data storage"
    Compliance     = "Enterprise"
    DataClass      = "Confidential"
  })
}

I used a project prefix combined with a random suffix to ensure global uniqueness while keeping bucket names recognizable.

Security Hardening

Public Access Block: Prevents accidental public exposure of bucket contents.

hljs hcl

resource "aws_s3_bucket_public_access_block" "secure_bucket_pab" {
  bucket = aws_s3_bucket.secure_bucket.id
  
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Server-Side Encryption: Implements AES-256 encryption for data at rest.

hljs hcl

resource "aws_s3_bucket_server_side_encryption_configuration" "secure_bucket_encryption" {
  bucket = aws_s3_bucket.secure_bucket.id
  
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

Versioning: Enables object versioning for data protection and recovery capabilities.

hljs hcl

resource "aws_s3_bucket_versioning" "secure_bucket_versioning" {
  bucket = aws_s3_bucket.secure_bucket.id
  
  versioning_configuration {
    status = "Enabled"
  }
}

Lifecycle Management

hljs hcl

resource "aws_s3_bucket_lifecycle_configuration" "secure_bucket_lifecycle" {
  bucket = aws_s3_bucket.secure_bucket.id
  
  rule {
    id     = "intelligent_tiering"
    status = "Enabled"
    
    filter {
      prefix = ""
    }
    
    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }
    
    transition {
      days          = 90
      storage_class = "GLACIER"
    }
    
    noncurrent_version_expiration {
      noncurrent_days = 90
    }
    
    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

This lifecycle configuration automatically transitions objects to cheaper storage classes over time: Standard IA after 30 days, Glacier after 90. Old versions get cleaned up, and incomplete multipart uploads are aborted after 7 days. This can reduce storage costs by up to 70%.

3. IAM Security Model

I implemented a least-privilege access model using IAM roles and policies -- no hardcoded credentials anywhere.

EC2 Instance Role (iam.tf)

hljs hcl

resource "aws_iam_role" "ec2_s3_role" {
  name = "${local.name_prefix}-ec2-s3-role"
  
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
  
  tags = local.common_tags
}

The trust policy restricts role assumption to EC2 instances only.

S3 Access Policy

hljs hcl

resource "aws_iam_role_policy" "ec2_s3_policy" {
  name = "${local.name_prefix}-ec2-s3-policy"
  role = aws_iam_role.ec2_s3_role.id
  
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket",
          "s3:GetBucketLocation",
          "s3:GetBucketVersioning",
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject",
          "s3:GetObjectVersion",
          "s3:DeleteObjectVersion"
        ]
        Resource = [
          aws_s3_bucket.secure_bucket.arn,
          "${aws_s3_bucket.secure_bucket.arn}/*"
        ]
      }
    ]
  })
}

The policy grants only the minimum S3 permissions needed, scoped specifically to the target bucket and its objects.

4. VPC Endpoints for Private Connectivity

VPC Endpoints keep S3 traffic on the AWS backbone instead of routing it through the internet. This improves both security and cost.

S3 VPC Endpoint (s3.tf)

hljs hcl

resource "aws_vpc_endpoint" "s3_endpoint" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${data.aws_region.current.name}.s3"
  vpc_endpoint_type   = "Gateway"
  route_table_ids     = [aws_route_table.private.id]
  
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = "*"
        Action = [
          "s3:ListBucket",
          "s3:GetBucketLocation",
          "s3:GetBucketVersioning",
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject",
          "s3:GetObjectVersion",
          "s3:DeleteObjectVersion"
        ]
        Resource = [
          aws_s3_bucket.secure_bucket.arn,
          "${aws_s3_bucket.secure_bucket.arn}/*",
          "arn:aws:s3:::amazonlinux-2-repos-${data.aws_region.current.name}",
          "arn:aws:s3:::amazonlinux-2-repos-${data.aws_region.current.name}/*"
        ]
      }
    ]
  })
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-s3-vpc-endpoint"
  })
}

Security Benefits:

Traffic remains within AWS network backbone
Eliminates exposure to internet-based attacks
Provides additional layer of access control through endpoint policies

Cost Benefits:

Reduces NAT Gateway data processing charges by ~78%
No hourly charges for Gateway endpoints
Lower latency for S3 operations

5. EC2 Infrastructure

Bastion Host Configuration (main.tf)

hljs hcl

resource "aws_instance" "bastion" {
  ami                         = data.aws_ami.amazon_linux.id
  instance_type               = var.instance_type
  key_name                    = var.key_pair_name
  subnet_id                   = aws_subnet.public.id
  vpc_security_group_ids      = [aws_security_group.bastion_sg.id]
  associate_public_ip_address = true
  
  user_data = base64encode(templatefile("${path.module}/user-data/bastion-userdata.sh", {
    hostname = "${local.name_prefix}-bastion"
  }))
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-bastion"
    Role = "Bastion"
  })
}

The bastion host provides the only SSH entry point into the private subnet.

Private Instance Configuration

hljs hcl

resource "aws_instance" "private_ec2" {
  ami                     = data.aws_ami.amazon_linux.id
  instance_type           = var.instance_type
  key_name                = var.key_pair_name
  subnet_id               = aws_subnet.private.id
  vpc_security_group_ids  = [aws_security_group.private_sg.id]
  iam_instance_profile    = aws_iam_instance_profile.ec2_profile.name
  
  user_data = base64encode(templatefile("${path.module}/user-data/private-userdata.sh", {
    hostname    = "${local.name_prefix}-private"
    bucket_name = local.bucket_name
    region      = data.aws_region.current.name
  }))
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-private"
    Role = "Application"
  })
}

This instance sits in the private subnet with no direct internet access and uses an IAM instance profile for S3 access instead of hardcoded credentials.

6. Security Groups

I configured stateful security groups following least-privilege principles.

Bastion Security Group

hljs hcl

resource "aws_security_group" "bastion_sg" {
  name_prefix = "${local.name_prefix}-bastion-sg"
  vpc_id      = aws_vpc.main.id
  description = "Security group for bastion host"
  
  ingress {
    description = "SSH from internet"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = var.allowed_ssh_cidrs
  }
  
  egress {
    description = "All outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-bastion-sg"
  })
}

SSH access is restricted to specified IP ranges only.

Private Instance Security Group

hljs hcl

resource "aws_security_group" "private_sg" {
  name_prefix = "${local.name_prefix}-private-sg"
  vpc_id      = aws_vpc.main.id
  description = "Security group for private EC2 instances"
  
  ingress {
    description     = "SSH from bastion"
    from_port       = 22
    to_port         = 22
    protocol        = "tcp"
    security_groups = [aws_security_group.bastion_sg.id]
  }
  
  egress {
    description = "HTTPS outbound"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  
  egress {
    description = "HTTP outbound"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-private-sg"
  })
}

The private instance only accepts SSH from the bastion security group and allows outbound HTTPS/HTTP for package updates and AWS API calls.

Audit and Monitoring Implementation

1. CloudTrail Configuration

I configured CloudTrail to capture API-level auditing for all service interactions, including object-level S3 operations.

CloudTrail Setup (cloudtrail.tf)

hljs hcl

resource "aws_cloudtrail" "main_trail" {
  name           = "${local.name_prefix}-cloudtrail"
  s3_bucket_name = aws_s3_bucket.cloudtrail_logs.bucket
  
  include_global_service_events = true
  is_multi_region_trail        = false
  enable_logging               = true
  
  event_selector {
    read_write_type                 = "All"
    include_management_events       = true
    
    data_resource {
      type   = "AWS::S3::Object"
      values = ["${aws_s3_bucket.secure_bucket.arn}/*"]
    }
  }
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-cloudtrail"
  })
}

Monitoring Scope:

Management Events: API calls for AWS service configuration changes
Data Events: Object-level operations on S3 bucket contents
Global Services: IAM, CloudFront, and other global service events

CloudTrail Log Storage

hljs hcl

resource "aws_s3_bucket" "cloudtrail_logs" {
  bucket = "${local.name_prefix}-cloudtrail-logs-${random_id.cloudtrail_suffix.hex}"
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-cloudtrail-logs"
    Purpose = "CloudTrail audit logs"
  })
}

Security Measures:

Separate bucket for audit logs to prevent tampering
Server-side encryption enabled
Public access blocked
Lifecycle policies for cost management

2. S3 Access Logging

S3 access logs provide detailed request-level information for security analysis and performance monitoring.

Access Log Configuration (s3-logging.tf)

hljs hcl

resource "aws_s3_bucket_logging" "secure_bucket_logging" {
  bucket = aws_s3_bucket.secure_bucket.id
  
  target_bucket = aws_s3_bucket.access_logs.id
  target_prefix = "access-logs/"
}

Log Information Captured:

Request timestamp and processing time
Client IP address and User-Agent
HTTP method and response code
Bytes transferred and object size
Authentication and authorization details

Access Log Storage Bucket

hljs hcl

resource "aws_s3_bucket" "access_logs" {
  bucket = "${local.name_prefix}-access-logs-${random_id.access_logs_suffix.hex}"
  
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-access-logs"
    Purpose = "S3 access logs"
  })
}

Cost Optimization: Lifecycle policies automatically transition logs to cheaper storage classes and delete old logs to manage storage costs.

Problems I Solved

1. Network Security

I isolated application workloads in a private subnet, used a bastion host for controlled access, and routed S3 traffic through VPC endpoints instead of the public internet. This eliminated direct internet exposure for the application tier.

2. Credential-Free Access

Instead of hardcoding AWS credentials, I used IAM roles with instance profiles. The EC2 instance assumes its role automatically -- no access keys to rotate or leak.

3. Data Protection

Every S3 object gets AES-256 encryption at rest. Versioning protects against accidental deletes, and CloudTrail plus S3 access logs create a complete audit trail.

4. Cost Control

Lifecycle policies automatically transition objects to Standard-IA after 30 days and Glacier after 90, cutting storage costs significantly. The S3 Gateway endpoint also eliminates NAT Gateway data processing charges for S3 traffic.

5. VPC Endpoint Policy Gotcha

This one caught me off guard. During testing, my VPC endpoint policy was too restrictive -- it blocked access to the Amazon Linux package repositories, so yum update failed on the private instance.

Root cause: VPC endpoint policies act as an additional access control layer on top of IAM. Even if IAM allows the action, the endpoint policy can still block it.

Fix:

hljs hcl

policy = jsonencode({
  Version = "2012-10-17"
  Statement = [
    {
      Effect = "Allow"
      Principal = "*"
      Action = [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject"
      ]
      Resource = [
        aws_s3_bucket.secure_bucket.arn,
        "${aws_s3_bucket.secure_bucket.arn}/*",
        "arn:aws:s3:::amazonlinux-2-repos-*",
        "arn:aws:s3:::amazonlinux-2-repos-*/*"
      ]
    }
  ]
})

Lesson learned: When writing VPC endpoint policies, you have to account for all S3 resources your instances need -- including system repos like amazonlinux-2-repos-*. This is easy to overlook.

Operational Procedures

1. Access Patterns

Administrative Access:

hljs bash

# Connect to bastion host
ssh -i keypair.pem ec2-user@<bastion-public-ip>

# Connect to private instance through bastion
ssh -A -i keypair.pem -o ProxyJump=ec2-user@<bastion-public-ip> ec2-user@<private-ip>

S3 Operations:

hljs bash

# List bucket contents
aws s3 ls s3://bucket-name/

# Upload files
aws s3 cp file.txt s3://bucket-name/path/

# Download files
aws s3 cp s3://bucket-name/path/file.txt ./

2. Monitoring and Analysis

CloudTrail Log Analysis:

hljs bash

# Download recent CloudTrail logs
aws s3 cp s3://cloudtrail-bucket/recent-log.json.gz ./

# Extract and analyze events
gunzip recent-log.json.gz
jq '.Records[] | select(.eventSource == "s3.amazonaws.com")' recent-log.json

S3 Access Log Analysis:

hljs bash

# Download access logs
aws s3 cp s3://access-logs-bucket/access-logs/recent.log ./

# Analyze request patterns
awk '{print $8}' recent.log | sort | uniq -c | sort -nr

Security Considerations

1. Data Classification

The implementation supports multiple data security levels:

Public: No restrictions (not implemented in this secure configuration)
Internal: Accessible within VPC through proper authentication
Confidential: Encrypted at rest and in transit, comprehensive audit logging
Restricted: Additional access controls and monitoring (future enhancement)

2. Compliance Framework Alignment

The architecture aligns with several compliance frameworks:

SOC 2: Comprehensive logging and access controls
ISO 27001: Risk-based security controls and continuous monitoring
NIST Cybersecurity Framework: Defense-in-depth implementation
GDPR: Data protection through encryption and access controls

3. Threat Mitigation

Threats Addressed:

Data Breaches: Encryption, access controls, and network isolation
Insider Threats: Least-privilege access and comprehensive audit logging
Configuration Drift: Infrastructure as Code ensures consistent configurations
Unauthorized Access: Multi-layer authentication and authorization

Cost Analysis

1. Infrastructure Costs (Monthly Estimates)

EC2 Instances: ~$30-50 (t3.micro instances)
NAT Gateway: ~$45 (fixed cost + data processing)
S3 Storage: Variable based on usage, optimized through lifecycle policies
CloudTrail: $2 per 100,000 events for data events
VPC Endpoints: Free for Gateway endpoints (S3, DynamoDB)

2. Cost Optimization Strategies

Storage Class Transitions: Automatic movement to cheaper storage classes
VPC Endpoints: Reduced NAT Gateway data processing charges
Log Lifecycle Policies: Automated deletion of old audit logs
Resource Tagging: Detailed cost allocation and optimization opportunities

Conclusion

This lab reinforced something I keep coming back to: security is not a single setting -- it is layers. Encryption, network isolation, least-privilege IAM, VPC endpoints, and audit logging each cover different threat vectors. The VPC endpoint policy issue was a good reminder that even well-planned configurations can have blind spots, and testing in the actual environment (not just terraform plan) is essential.

Future Enhancements

Potential improvements to consider:

Multi-AZ deployment for high availability
AWS Config for configuration compliance monitoring
AWS Security Hub for centralized security findings management
AWS GuardDuty for threat detection and monitoring
Cross-region replication for disaster recovery
AWS KMS for customer-managed encryption keys
AWS Systems Manager for patch management and configuration
Amazon Macie for data discovery and classification