Post

VPC

Posted
By Hector G Serrano 18 min read

We first start with some terminology:

  • VPC - Virtual Private Cloud
  • CIDR - Classless Inter-Domain Routing
  • ENI - Elastic Network Interface
  • SG - Security Group
  • BYOI - Bring Your Own IP

VPC

By default you can create up to 5 VPCs per region. See: Amazon VPC quotas

You can ask for additional VPCs per region using:

Old mechanisms to communicate between different VPCs:

  • VPC peering
  • Through Internet

Scope:

  • VPCs are region specifics
  • One VPC spans Availability Zones (AZs)

Services in VPC:

  • EC2
  • RDS
  • Redshift
  • Elastic Load Balancing
  • etc…

Services NOT inside a VPC:

  • S3
  • DynamoDB
  • Lambda
  • API Gateway
  • SQS
  • SNS
  • etc…

CIDR

  • When you create a VPC, you must specify an IPv4 CIDR block for the VPC
  • The allowed block size is between a /16 netmask and /28 netmask
  • After you’ve created your VPC, you can associate additional IPv4 CIDR blocks with the VPC
  • You cannot increase or decrease the size of an existing CIDR block
  • The CIDR block of a subnet can be:
    • the same as the CIDR block for the VPC (for a single subnet in the VPC)
    • a subset of the CIDR block for the VPC (to create multiple subnets in the VPC)

Secondary CIDRs can be added to an existing VPC

  • CIDR block must not overlap with existing CIDR or peered VPC CIDR
  • If Primary CIDR is from RFC 1918 then you can not add secondary CIDR from other RFC1918 IP ranges (10.0.0.0/18, 172.16.0.0/12, 192.168.0.0/16)
  • CIDR block must not be same or larger than the CIDR range from routes in any of the VPC Route tables
  • You can have total 5 IPv4 and 1 IPv6 CIDE block for VPC

Documentation:

Subnets, Route Tables and Internet Gateway

  • The scope of the Internet Gateway is VPC level (it does not belong to a specific AZ)
  • Subnets belong only to a single AZ
  • EC2 instances are attached to a single subnet and, therefore attached to a single AZ
  • Subnets are attached to a Route table
    • A subnet can be attached only to a single Route table
    • However, a Route table can have multiple subnets attached to it

The diagram shows two EC2 instances on different AZs. Both Subnets are attached to the Main Route table so communication between both EC2 instances is possible.

To access the Internet, an Internet Gateway is required.

Public Subnet vs Private Subnet

In addition to the Main Route table, a Custom Route table can optionally be created. In this diagram, since the Custom Route table does not have a 0.0.0.0/0 route to the Internet Gateway (IGW), the EC2 instance with IP address 172.31.1.11 cannot have access to the Internet.

If the subnet has a 0.0.0.0/0 to the IGW it is a Public Subnet. Otherwise, a subnet with no internet access is called in AWS terms Private subnet.

IP Addressing

5 IP addresses cannot be used per subnet.

  • 172.31.0.0/24 - Network address
  • 172.31.0.1/24 - Reserved by AWS for the VPC Router
  • 172.31.0.2/24 - Reserved by AWS for mapping to Amazon-provided DNS
  • 172.31.0.3/24 – Reserved by AWS for future use
  • 172.31.0.255/24 - Broadcast address. AWS does not support broadcast in a VPC. This address cannot be used.

Same explanation from Subnet CIDR blocks:

The first four IP addresses and the last IP address in each subnet CIDR block are not available for your use, and they cannot be assigned to a resource, such as an EC2 instance. For example, in a subnet with CIDR block 10.0.0.0/24, the following five IP addresses are reserved:

  • 10.0.0.0: Network address.
  • 10.0.0.1: Reserved by AWS for the VPC router.
  • 10.0.0.2: Reserved by AWS. The IP address of the DNS server is the base of the VPC network range plus two. For VPCs with multiple CIDR blocks, the IP address of the DNS server is located in the primary CIDR. We also reserve the base of each subnet range plus two for all CIDR blocks in the VPC. For more information, see Amazon DNS server.
  • 10.0.0.3: Reserved by AWS for future use.
  • 10.0.0.255: Network broadcast address. We do not support broadcast in a VPC, therefore we reserve this address.

EC2 instance Internet access

  1. An Internet Gateway (IGW)
  2. Subnet attached to a Route Table with 0.0.0.0/0 route pointing to the IGW
  3. A “Public IP address” assigned to the EC2 instance.

The Public IP address is assigned randomly by AWS to the EC2 instance from a pool of available IP addresses.

Option at Subnet level to auto-assign a PUblic IP address to the EC2 instance upon creation.

Private IPv4 vs Public IPv4 vs IPv6 addressing

Public IP addresses change upon stop/start of the EC2 instance. From Instance lifecycle

ENI - Elastic Network Interfaces

Local component in a VPC that represents a virtual network card. The ENI can have the following attributes:

  • Primary private IPv4, one or more secondary IPv4
  • One Elastic IP (IPv4) per private IPv4
  • One Public IPv4
  • One or more Security Groups (SG)
  • A MAC address

  • A source/destination check flag

  • Technically Security Groups (SGs) are attached to ENIs, not to EC2 instances
  • You can create ENI independently and attach them on the fly (move them) on EC2 instances for failover
  • When creating an ENI you must choose a Subnet. This bound ENIs to a specific AZ
  • The primary ENI cannot be detached from the EC2 instance
  • Second ENI allows instance to be multi-homed (on different Subnets) in the same AZ
  • ENIs can not be used for NIC teaming which means they can not be used together to increase instance network bandwidth

Maximum number of network interfaces per instance type is documented here:

All ENIs and EC2 instances must be in the same AZ.

DNS options - VPC level

Default configuration

DNS resolution

The DNS resolution attribute determines whether DNS resolution through the Amazon DNS server is supported for the VPC.

DNS hostnames

The DNS hostnames attribute determines whether instances launched in the VPC receive public DNS hostnames that correspond to their public IP addresses.

Auto-assign public IPv4 address - Subnet level

Automatically request a public IPv4 address for instances launched into this subnet.

Default configuration:

Enable auto-assign public IPv4 address By default, nondefault subnets have the IPv4 public addressing attribute set to false, and default subnets have this attribute set to true. An exception is a nondefault subnet created by the Amazon EC2 launch instance wizard — the wizard sets the attribute to true.

Transfer files to your Linux instance using the PuTTY Secure Copy client

1

pscp -i C:\path\my-key-pair.ppk C:\path\Sample_file.txt instance-user-name@instance-public-dns-name:/home/instance-user-name/Sample_file.txt

Transfer files to your Linux instance using the PuTTY Secure Copy client

Example:

1
2
3
4

C:\Users\gusta>pscp -i C:\Users\gusta\Documents\workspace_blog\gustavos86.github.io\images\2023\09-25-VPC\vpcpubhol.ppk C:\Users\gusta\Documents\workspace_blog\gustavos86.github.io\images\2023\09-25-VPC\vpcprivhol.pem ec2-user@ec2-34-235-171-126.compute-1.amazonaws.com:/home/ec2-user
vpcprivhol.pem            | 1 kB |   1.8 kB/s | ETA: 00:00:00 | 100%

C:\Users\gusta>

TIP: Convert .ppk to .pem format

.ppk is supported by PuTTY on Windows .pem is supported by Linux and Mac OS X

Source: How to Convert PPK to PEM file using Command

I was unable to download PuTTY on AMI Linux 2023.

1
2
3
4
5

[ec2-user@ip-XX-X-X-XX ~]$ sudo yum install putty
Last metadata expiration check: 0:29:33 ago on Wed Sep 27 05:18:26 2023.
No match for argument: putty
Error: Unable to find a match: putty
[ec2-user@ip-XX-X-X-XX ~]$

Found this as the reason:

SSH using .pem key

Do not forget to set low privileges to the .pem key.

1
2
3
4
5
6
7
8
9
10
11
12
13

[ec2-user@ip-XX-X-X-XX ~]$ chmod 600 vpcprivhol.pem
[ec2-user@ip-XX-X-X-XX ~]$ ssh -i vpcprivhol.pem ip-10-0-2-72.ec2.internal
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
[ec2-user@ip-10-0-2-72 ~]$

VPC Peering

Diagram:

It is configured in VPC > Peering Connection:

Security Groups

  • Operates at instance level (applied to ENI attached to EC2 instance)
  • Supports Allow rules only
  • Is stateful: Return trafffic is automatically allowed
  • All rules are evaluated before deciding whether to allow traffic
  • Can not block the traffic from specific IP address or CIDR block

Network Access Control List (NACL)

  • Operates at Subnet level
  • Supports both Allow and Deny rules
  • Is stateless: Return traffic must be explicitly allowed by rules
  • Rules are processed in number order and if matched no further rules are evaluated
  • With Deny rules, trafffic can be blocked from specific IP addresss or CIDR block

Default VPC

  • CIDR 172.31.0.0/16
  • One Subnet in every AZ with CIDR /20
  • All Subnets are attached to the Main Route Table
  • Main Route Table has a 0.0.0.0/0 to an Internet Gateway
  • If deleted, you can recreate the Default VPC
  • AWS Management Console > VPC Service > Your VPCs > Action > Create Default VPC

NAT Gateway

  • The scope of NAT is AZ level. This means the NAT Gateway belongs to a specific AZ
  • The NAT Gateway is a managed AWS service
  • It is highly available within the AZ
  • The NAT Gateway must be configured in a public subnet
  • Pay by the hour for usage and bandwidth
  • It is created in a specific AZ, uses an EIP
  • 5 Gbps of bandwidth with automatic scaling up to 100 Gbps
  • No Security Group to manage/requred. NACL at subnet level applies to NAT Gateway
  • Supported protocols: TCP, UDP, and ICMP
  • Uses ports 1024-65535 for outbound connection

NAT Instance

  • NAT EC2 can be launched using Amazon Linux NAT AMI (optionally). I think any EC2 instance with a properly configured OS works.
  • Disable Soruce/Destination check on instance
  • Must have a Public or Elastic IP
  • Must be in Public Subnet

NOTE: About assigning a Public IP to an EC2 instance which was not configured with having one assigned during creation. You must assign an Elastic IP in this case since you cannot assign an “ephemeral” Public IP once an EC2 instance has been launched without one. Also, I think we cannot change the subnet association of an EC2 instance’s ENI.

BYOI - Bring Your Own IP

  • Migrate your publicly routable IPv4 and IPv6 IP addresses to AWS
  • The address range must be registered with regional internet registry (RIR) - ARIN or RIPE or APNIC
  • The addresses in the IP address range must have a clean history
  • The most specific IPv4 address range that you can bring is /24
  • The most specific IPv6 address range that you can bring is /48 for CIDRs that are publicly advertised
  • The most specific IPv6 address range that you can bring is /56 for CIDRs that are not publicly advertised (can be advertised over Direct Connect if required)
  • Create a Route Origin Authorization (ROA) to authorize Amazon ASNs, 16509 and 14618 to advertise your address range
  • You continue to own the address range, but AWS advertises it on the internet in your behalf
  • After bringing the address range to AWS, it appears in your AWS account as an address pool
  • You can associate these IP addresses to Amazon EC2 instances, Network Load Balancers, and NAT gateways
  • You can bring a total of five IPv4 and IPv6 address ranges per Region to your AWS account

VPC Flow Logs

  • VPC Flow Logs can be enabled per:
    • ENI - Elastic Network Internet
    • Subnet
    • VPC
  • You can log your VPC Flog Logs to:
    • CloudWatch Logs
    • S3 bucket
    • Kinesis Firehose in the same account
    • Kinesis Firehose in a different account
  • VPC Flow Logs are often sent to CloudWatch. An alarm can be set based on specific metrics in the Flow Logs (example: traffic from specific ip addresses or CIDR blocks)
  • Once in CloudWatch, the logs can also be exported to S3 for more cost effective storage
  • Bug VPC Flow Logs can now be sent directly to S3 which enables them to be queried by Athena

DHCP Option Set

  1. Create a new DHCP option set in the VPC section

This image shows a DHCP option set already created with a custom Domain name and Domain name servers

  1. Attach a the new DHCP option set to your VPC

Select the desired VPC in Your VPCs section.

Click on the Actions button located in the top-right side, select Edit VPC settings.

Chose the new DHCP option set for this VPC.

  • Once created, you can not modify DHCP Options set. However you can create a ne DHCP Options set and associate it with the VPC
  • You can only associate a single DHCP Options set to a VPC
  • VPC could have no DHCP Options set associated
  • After DHCP Options set is associated with a VPC, the EC2 instances automatically use the new option set, but this may take a few hours.
  • Refreshing the DHCP option parametesr on the EC2 instance is possible running the command:
1

#sudo dhclient -r eth0

AWS assigned domain names for EC2

  • Internal DNS
    • ip-private-ipv4-address.ec2.internal (for us-east-1 region)
    • ip-private-ipv4-address.region.compute.internal (for all other regions)
  • External DNS (if EC2 instance has Public IP)
    • ec2-public-ipv4-address.compute-1.amazonaws.com (for us-east-1 region)
    • ec2-public-ipv4-address.region.compute.amazonaws.com (for all other regions)

DNS

  • enableDnsSupport: (this is DNS Resolution setting)
    • Default True
    • Helps decide if DNS resolution is supported in the VPC
    • If True, queries the AWS DNS server at 169.254.169.253 virtual IP (same as VPC base address plus 2)
  • enableDnsHostname: (this is DNS Hostname setting)
    • False by default for newly created VPC, True by default for Default VPC
    • Won’t do anything unless enableDnsSuport is True
    • If True, assigns public histname to EC2 instance if it has a public IP
  • If you use custom DNS domain names defined in a private hosted zone in AWS Route 53, you must set both the enableDnsHostname & enableDnsSupport attribute to True

You can also query DNS server at 169.254.169.253

@@ Defining MTU on EC2 instances

  • MTU depends on Instance Type
  • Defined at ENI level
  • tracepath amazon.com to check the path MTU between EC2 instance and target endpoint
  • ip link show eth0 to check the MTU on the interface
  • sudo ip link set dev eth0 mtu 9001 to set the MTU value

MTU within AWS:

  • Within VPC: supports Jumbo frames (9001 bytes)
  • Over the VPC Endpoint: MTU 8500 bytes
  • Over the Internet Gateway (IGW): MTU 1500 bytes
  • Intra region VPC Peering: MTU 9001 bytes
  • Inter region VPC Peering: MTU 1500 bytes

On-premise network:

  • Over the VPC using VGW: MTU 1500
  • Over the VPC via Transit Gateway: MTU 1500 for traffic for Site-to-Site VPN
  • Over the DirectConnect (DX): Supports Jumbo frames (9001 bytes)
  • Over the DX via Transit Gateway: MTU 8500 for VPC attachments connected over the Direct Connect

Cluster Placement Groups

Group of EC2 instances launches togehther, as physical as close to each other.

  • Logical grouping of instances within the same AZ
  • Ideal for High Performance Computing

EBS Optimized Instances

Delivers dedicated throughput between EC2 instances and Amazon EBS

  • EBS is a network drive (i.e. not a physical drive)
  • Minimizes contention between Amazon EBS Input/Output (I/O) and othe traffic from Amazon EC2 instance

Enhanced Networking (Intel vs ENA)

Depending on EC2 instance type, Enhanced Networking can be enabled using one of the following network drivers:

  • Option I: Intel 82599 VF up to 10 Gbps (VF uses ixgbevf driver)
  • Option II: Elastic Network Adapter (ENA) up to 100 Gbps (This 100G is with combined flows. Still 10Gbps per flow even when the EC2 intances are part of the same Placement Group) The elegible EC2 instance families support either of the above two drivers

For Enhanced Networking, it requires support from both EC2 OS (AMI) and Instnace Type what is flagged for Enhanced Networking

Verifying enhanced networking in the EC2 instance

1

$ethtool -i eth0

DPDK & EFA

DPDK - Intel Data Plane Development Kit

Application needs to be developed unsing DPDK to bypass the Kernel and send the traffic directly to the network driver.

  • DPDK is a set of libraries and drivers for fast packet processing.
  • While Enhanced Networking and SR-IOV reduce overhead of packet processing between the EC2 Instance and the Hypervisor, DPDK reduces overhead of packet processing inside the Operating System.
  • DPDK provides:
    • Lower latency due to Kernel bypass
    • More control of packet processing
    • Lower CPU overhead

EFA - Elastic Fabric Adapter

  • EFA is an ENA with added capabilities
  • Provides lower latency and higher throughput
  • Provides OS bypass functionality (Linux). EFA can only be used in Linux machines.
  • For Windows instances, it acts just as ENA.
  • With an EFA, HPC applications use MPI to interface with the Libfabric API which bypasses OS kernel and communicates directly with the EFA device to put packets on the network.

Bandwidth Limits

  • VPC. No specific limits.
  • Internet Gateway (IGW). No specific limits.
  • VPC peering. No specific limits.

  • NAT Gateway. Can provide up to 45 Gbps. Use multiple NAT Gateways to scale beyond 45 Gbps. Make sure traffic does not flow across AZs as this incurs additional charges.

  • EC2 depends on factors like Instance family, vCPU, traffic destination, etc
    • If traffic destination is within the Region, the maximum bandwidth listed for the EC2 instance should apply
    • If traffic is outside is the outside the Region, up to 50% of the network bandwidth (current generation instance with a minimum of 32 vCPUs)
    • If traffic is outside is the outside the Region, limited to 5 Gbps on older EC2 generation
  • EC2 with Intel 82599 VF interface. 10 Gbps aggregate and 5 Gbps flow-based bandwidth limit
  • AWS ENA driver.
    • 10 Gbps flow limit inside a placement group
    • 5 Gbps flow limit outside of a placement group
    • Aggregate bandwidth of 100 Gbps with multiple flows within a VPC or a peered VPC or to S3 (using VPC endpoint) in the same region
  • Virtual Private Gateway (VGW). 1.25 Gbps aggregate bandwidth per VGW for traffic from AWS to on-premises
    • Multiple VPN connections to the same VGW are bound by an aggregate throughput limit
  • AWS Direct Connect bandwidth is defined by Port Speed
    • For AWS Direct Connect connection on a VGW, the throughput is bound by the Direct Connect port itself
  • Transit Gateway supports 1.25 Gbps per VPN tunnel and 50 Gbps total VPN bandwidth

Network I/0 Credits

  • Instance families such as R4 and C5 use a network I/O credit mechanism
  • Most applications do not consistently need a high network performance
  • These instances perform well above baseline n/w performance during peak requirement
  • Make sure that you consider the accumulated n/w credits before doing performance benchmark for instances supporting network I/O credits mechanism

Network Performance and Optimization essentials

  • Within VPC the MTU size can be 9001 bytes with Jumbo Frames
  • For the following cases maximum MTU is 1500 bytes:
    • Traffic over an Internet Gateway
    • Traffic over an inter-region VPC peering connection
    • Traffic over VPN connections
  • When PPS is bottleneck, increased MTU provides more throughput

  • While Enhanced Networking and Placement Groups lowers the latency between EC2 and Hypervisor, DPDK improves the packet processing at OS level

  • EC2 network bandwidth depends on many factors including Instance family, size, enhance networking support, etc
  • Bandwidth for aggregate multi-flow traffic available to an instance depends on the destination of the traffic
    • Within the Region
      • Full bandwidth. Check Amazon EC2 instance network bandwidth for more specifics
      • Within the AWS region you can get up to 100 Gbps bandwidth between EC2 instances or between EC2 and S3 by using multiple-flows
    • Outside of the Region, Internet or Direct Connect
      • 50% of the available network bandwidth for current generation instances with minimum 32 vCPUs
      • 5 Gbps for instances with less than 32 vCPU

    Bandwidth for single-flow (5-tuple) traffic is limited to 5 Gbps and can get up to 10 Gbps if the instances are within the Cluste replacement group

AWS, VPC
aws vpc
This post is licensed under CC BY 4.0 by the author.