AWS Setup

This guide walks you through preparing your AWS environment for EZ-CDC deployment.

Overview

EZ-CDC workers run in your AWS account and need:

1. A VPC with private subnets
2. NAT Gateway or VPC endpoints for outbound connectivity
3. Security groups for network access control
4. An IAM role for worker permissions

VPC Configuration

VPC Requirements

Your VPC must have:

• DNS support and DNS hostnames enabled
• At least 2 private subnets in different Availability Zones
• NAT Gateway (or VPC endpoints) for outbound internet access
• No inbound access required — workers only make outbound connections

Required VPC Settings

# DNS settings must be enabled
resource "aws_vpc" "main" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_support   = true   # Required for endpoint resolution
  enable_dns_hostnames = true   # Required for private DNS
}

Subnet Requirements

Subnet Type	Purpose	Internet Access
Private	Worker instances	Via NAT Gateway
Public	NAT Gateway	Direct

NAT Gateway Setup

Workers in private subnets need NAT Gateway for outbound connectivity:

# Elastic IP for NAT
resource "aws_eip" "nat" {
  domain = "vpc"
}

# NAT Gateway in public subnet
resource "aws_nat_gateway" "main" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public.id
}

# Route table for private subnets
resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.main.id
  }
}

# Associate with private subnets
resource "aws_route_table_association" "private" {
  subnet_id      = aws_subnet.private.id
  route_table_id = aws_route_table.private.id
}

Security Groups

Worker Security Group

Create a security group for EZ-CDC workers:

resource "aws_security_group" "ezcdc_worker" {
  name        = "ezcdc-worker"
  description = "EZ-CDC Worker Agent security group"
  vpc_id      = aws_vpc.main.id

  # NO INBOUND RULES - workers don't accept connections

  # Outbound: HTTPS for control plane and S3
  egress {
    description = "HTTPS outbound"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # Outbound: PostgreSQL (adjust CIDR to your source)
  egress {
    description     = "PostgreSQL source"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.postgres.id]
  }

  # Outbound: StarRocks MySQL protocol
  egress {
    description     = "StarRocks MySQL"
    from_port       = 9030
    to_port         = 9030
    protocol        = "tcp"
    security_groups = [aws_security_group.starrocks.id]
  }

  # Outbound: StarRocks HTTP (Stream Load)
  egress {
    description     = "StarRocks HTTP"
    from_port       = 8040
    to_port         = 8040
    protocol        = "tcp"
    security_groups = [aws_security_group.starrocks.id]
  }

  tags = {
    Name = "ezcdc-worker"
  }
}

Database Security Groups

Allow workers to connect to your databases:

# PostgreSQL security group rule
resource "aws_security_group_rule" "postgres_from_worker" {
  type                     = "ingress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.ezcdc_worker.id
  security_group_id        = aws_security_group.postgres.id
  description              = "PostgreSQL from EZ-CDC workers"
}

# StarRocks security group rules
resource "aws_security_group_rule" "starrocks_mysql_from_worker" {
  type                     = "ingress"
  from_port                = 9030
  to_port                  = 9030
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.ezcdc_worker.id
  security_group_id        = aws_security_group.starrocks.id
  description              = "StarRocks MySQL from EZ-CDC workers"
}

resource "aws_security_group_rule" "starrocks_http_from_worker" {
  type                     = "ingress"
  from_port                = 8040
  to_port                  = 8040
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.ezcdc_worker.id
  security_group_id        = aws_security_group.starrocks.id
  description              = "StarRocks HTTP from EZ-CDC workers"
}

VPC Endpoints (Optional)

For PrivateLink deployments or to reduce NAT costs:

# S3 Gateway Endpoint (free, recommended)
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]
}

# SSM endpoints (for Session Manager access)
resource "aws_vpc_endpoint" "ssm" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ssm"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.vpce.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "ssmmessages" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ssmmessages"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.vpce.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "ec2messages" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ec2messages"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private.id]
  security_group_ids  = [aws_security_group.vpce.id]
  private_dns_enabled = true
}

Verification

After setup, verify your configuration:

1. Check VPC DNS Settings

aws ec2 describe-vpc-attribute \
  --vpc-id vpc-xxxxx \
  --attribute enableDnsSupport

aws ec2 describe-vpc-attribute \
  --vpc-id vpc-xxxxx \
  --attribute enableDnsHostnames

Both should return true.

2. Check NAT Gateway

aws ec2 describe-nat-gateways \
  --filter Name=vpc-id,Values=vpc-xxxxx

Should show state available.

3. Check Route Tables

aws ec2 describe-route-tables \
  --filters Name=vpc-id,Values=vpc-xxxxx

Private subnets should route 0.0.0.0/0 to NAT Gateway.

Next Steps

With AWS configured:

1. Create IAM Role - Set up permissions
2. Create Deployment - Launch workers