You may not be an expert on container networking, but wouldn't you like to impress guests at your next party by explaining the difference between "host" and "bridge" networking?
This week on Mobycast, Jon and Chris conclude their three-part series on serverless containers with AWS Fargate. We wrap our heads around container networking and its various networking modes, with particular emphasis on task networking (aka "awsvpc" mode).
We finish by pulling together everything we learned over these 3 episodes to walk step-by-step through the migration of a container from EC2 to Fargate. After this episode, you'll be the life of the party!
Show Notes
Show Details
In this episode, we cover the following topics:
- Container networking
- ECS networking mode
- Configures the Docker networking mode to use for the containers in the task
- Specified as part of the task definition
- Valid values:
- none
- Containers do not have external connectivity and port mappings can't be specified in the container definition
- bridge
- Utilizes Docker's built-in virtual network which runs inside each container instance
- Containers on an instance are connected to each other using the docker0 bridge
- Containers use this bridge to communicate with endpoints outside of the instance using primary ENI of instance they are running on
- Containers share networking properties of the primary ENI, including the firewall rules and IP addressing
- Containers are addressed by combination of IP address of primary ENI and host port to which they are mapped
- Cons:
- You cannot address these containers with the IP address allocated by Docker
- It comes from pool of locally scoped addresses
- You cannot enforce finely grained network ACLs and firewall rules
- host
- Bypass Docker's built-in virtual network and maps container ports directly to the EC2's NIC directly
- You can't run multiple instantiations of the same task on a single container instance when port mappings are used
- awsvpc
- Each task is allocated its own ENI and IP address
- Multiple applications (including multiple copies of same app) can run on same port number without conflict
- You must specify a NetworkConfiguration when you create a service or run a task with the task definition
- Default networking mode is bridge
- host and awsvpc network modes offer the highest networking performance
- They use the Amazon EC2 network stack instead of the virtualized network stack provided by the bridge mode
- Cannot take advantage of dynamic host port mappings
- Exposed container ports are mapped directly...
- host: to corresponding host port
- awsvpc: to attached elastic network interface port
- Task networking (aka awsvpc mode networking)
- Benefits
- Each task has its own attached ENI
- With primary private IP address and internal DNS hostname
- Simplifies container networking
- No host port specified
- Container port is what is used by task ENI
- Container ports must be unique in a single task definition
- Gives more control over how tasks communicate
- With other tasks
- Containers share a network namespace
- Communicate with each other over localhost interface
- With other services in VPC
- Note: containers that belong to the same task can communicate over the localhost interface
- Take advantage of VPC Flow Logs
- Better security through use of security groups
- You can assign different security groups to each task, which gives you more fine-grained security
- Limitations
- The number of ENIs that can be attached to EC2 instances is fairly small
- E.g. c5.large EC2 may have up to 3 ENIs attached to it
- 1 primary, and 2 for task networking
- Therefore, you can only host 2 tasks using awsvpc mode networking on a c5.large
- However, you can increase ENI density using "VPC trunking"
- VPC trunking
- Allows for overcoming ENI density limits
- Multiplexes data over shared communication link
- How it works
- Two ENIs are attached to the instance
- Primary ENI
- Trunk ENI
- Note that enabling trunking consumes an additional IP address per instance
- Your account, IAM user, or role must opt in to the awsvpcTrunking account setting
- Benefits
- Up to 5x-17x more ENIs per instance
- E.g. with trunking, c5.large goes from 3 to 12 ENIs
- 1 primary, 1 trunk, and 10 for task networking
- Migrating a container from EC2 to Fargate
- IAM roles
- Roles created automatically by ECS
- Amazon ECS service-linked IAM role, AWSServiceRoleForECS
- Gives permission to attach ENI to instance
- Task Execution IAM Role (ecsTaskExecutionRole)
- Needed for:
- Pulling images from ECR
- Pushing logs to CloudWatch
- Create a task-based IAM role
- Required because we don't have an ecsInstanceRole anymore
- Create a IAM policy that gives minimal privileges needed by task
- Remember two categories of policies:
- AWS Managed
- Customer Managed
- We are going to create a new customer managed policy that contains only the permissions our app needs
- KMS Decrypt, S3 GETs from specific bucket
- IAM -> Policies -> Create Policy -> JSON
- Create role based on "Elastic Container Service Task" service role
- This service role gives permission to ECS to use STS to assume role (sts:AssumeRole) and perform actions on its behalf
- IAM -> Roles -> Create Role
- "Select type of trusted entity": AWS Service
- Choose "Elastic Container Service", and then "Elastic Container Service Task" use case
- Next, then attach IAM policy we created to the role and save
- Task definition file changes
- Task-level parameters
- Add FARGATE for requiredCompatibilities
- Use awsvpc as the network mode
- Specify cpu and memory limits at the task level
- Specify Task Execution IAM Role (executionRoleARN)
- Allows task to pull images from ECR and send logs to CloudWatch Logs
- Specify task-based IAM role (taskDefinitionArn)
- Needed to give task permissions to perform AWS API calls (such as S3 reads)
- Container-level parameters
- Only specify containerPort (do not specify hostPort)
- See Task Definition example below
- Create ECS service
- Choose cluster
- Specify networking
- VPC, subnets
- Create a security group for this task
- Security group is attached to the ENI
- Allow inbound port 80 traffic
- Auto-assign public IP
- Attach to existing application load balancer
- Specify production listener (port/protocol)
- Create a new target group
- When creating target group, you specify "target type"
- Instance
- IP
- Lambda function
- For awsvpc mode (and by default, Fargate), you must use the IP target type
- Specify path pattern for ALB listener, health check path
- Note: you cannot specify host-based routing through the console
- You can update that after creating the service through the ALB console
- Update security groups
- Security group for ALB
- Allow outbound port 80 to the security group we attached to our ENI
- Security group for RDS
- Allow inbound port 3306 from the security group for our ENI
- Create Route 53 record
- ALIAS pointing to our ALB
- Log integration with SumoLogic
- Update task to send logs to stdout/stderr
- Configure containers for CloudWatch logging ("awslogs" driver)
- Create Lambda function that subscribes to CloudWatch Log Group
- Lambda function converts from CloudWatch format to Sumo, then POSTs data to Sumo HTTP Source
- DeadLetterQueue is recommended to handle/retry failed sends to Sumo
Links
For a full transcription of this episode, please visit the
episode webpage.
We'd love to hear from you! You can reach us at:
What is Mobycast?
A Podcast About Cloud Native Software Development, AWS, and Distributed Systems