The goal is to avoid going to the internet and fetching everything from the AWS backbone. This is the AWS recommended solution, keep in mind that it is complex to configure, maintain, and costly
Last updated
For configuring cross-region access for AWS services without leaving the AWS Backbone first of all we need to follow the steps of leveraging endpoints for the console region mentioned in our documentation. Then we need to execute the same script in the desired region to interact with, allow communication through VPC endpoints and setup DNS resolution so the public url service endpoints get resolved to local ips instead of the public ones.
Console Region: us-west-2
Agent Region: us-east-2
AWS lines up what is needed in order to accomplish this task in the following two documents:
https://repost.aws/knowledge-center/vpc-endpoints-cross-region-aws-services: despite it says one or the other both sections are needed.
Above tutorials contain a lot of suppositions and don’t have the level of detail necessary to actually understand the needed changes. The following are some steps done in our development accounts in order to accomplish cross region:
Apply CSS the CSS provided scripts about leveraging VPC endpoints in the console’s and destination region.
Configure VPC Peering Connections between the console and destination region VPCs. Ensure DNS settings are enabled for both:
Configure route tables for both VPCs, in each of them add an entry containing the CIDR of the other region’s VPC and as destination the Peering connection:
Validate no subnet contains access to a nat or internet gateway, meaning they can’t reach the internet. Create an Ubuntu instance that is in the Console’s region and subnet. Connect to it, probably using a bastion host (public instance within the same VPC but different subnet that can reach the private subnet).
So right now you are in the console’s subnet, validate you are able to reach the EC2 service belonging to the other destination region (in this case us-east-2) by running the following command with valid AWS Credentials:
Keep in mind the usage of the destination region and the local private IP of the EC2 VPC Endpoint of the us-east-2 region. This is necessary since there is no DNS resolution configured for the endpoint “ec2.us-east-2.amazonaws.com”.
Note also that without the endpoint specification the EC2 instance doesn’t know how to solve the service URL:
At this point we are able to reach the VPC Endpoint located in the destination region but without being able to automatically solve the EC2 service. Let’s get into it.
Go to route 53 and add a new private hosted zone adding the console’s VPCs.
Add an A record with Alias selected and the destination region VPC Endpoint as the target:
Validate configuration by doing the Ec2 API call without endpoint specification:
Services that require this extra configuration since they are instantiated by the console in other regions:
EC2: For Networking stuff
ElasticFileSystem: To mount volumes
CloudWatchLogs: To build statistic results, protecting a bucket (creates CloudStorageSecurity.ECS.{EnvironmentVariables.AppConfigAgentApplicationId}.{serviceDescription} and handles retention policy)
CloudWatch (called monitoring): To create alarms.
S3 (interface + gateway): To gather bucket properties. Remember to add a wildcard for the global bucket access:
SNS: To list topics
ECS: For ECS related tasks, like creating a cluster or a service
AppAutoScaling: for autoscaling policies triggered from CW Alarms
VPC Endpoints in the Agent Region
VPC Endpoints in the Console Region
Route53 custom endpoints configuration
The agent task running in the external region needs a proxy defined to reach DynamoDB or internet access to allow its resolution.
SNS notifications for the Agent will fail since it doesn’t know how to resolve the Console’s region SNS URL endpoint. A route53 entry resolution is needed for the console region associated with the agent region VPC and pointing to resolution for the VPC SNS Endpoint of the console. Also remember to add a rule to the Security Grouping allowing the VPC CIDR of the agent.
Using a private subnet allowing only a few services (Marketplace, AppConfig and Cognito) to go through the public internet (maybe behind a proxy) and then all the other services being accessed through VPC Endpoints is definitely doable, even for all the regions.
Pros:
Great reduction in the amount of traffic over the public internet.
Cons:
Increased infrastructure costs.
Increased infrastructure complexity, harder to configure and to maintain.
New AWS services usage from our part can break console functionality.
Costs: only the VPC piece cost around $10 daily for the Console region and $7 daily for each extra region.
