Skip to content

API Agent Settings

Agent Settings The API Agent Settings page is used to setup API endpoint regions and make modifications to the agent task characteristics. The API scanning agent can be deployed to most regions at this time. Unlike the Event Agent which is setup at the time protection is turned on for a bucket in a new region, the API agent is specifically setup in regions of your choice completely independently of whether you have S3 integrated protections (event-based and retro-based scanning) in place. As part of the deployment process, a Load Balancer is created as the entry point to the API Agent. After setup is complete, you will have an API Agent Service in your Fargate Cluster and a Load Balancer (either internet-facing or internal) as the persistent access mechanism. On top of this setup, you can configure aspects of the scanning agent and characteristics for how they scale. This includes the CPU, Memory and Disk Size (for the AWS Fargate tasks), the VPC and Subnet(s) as well as the scaling characteristics including Minimum # of Agents and Maximum # of Agents.

With the API Agent you have choices as to whether you deploy in more than one region. With the Event Agent, we deployed infrastructure to each region with protected buckets to keep the scanning close to the data. With the API Agent effectively being an API endpoint, you could make one bit of infrastructure available to all of your applications if networking and latency allowed for it. Applications, whether on-prem, living in AWS or being run by third parties could leverage one endpoint. You have the choice to deploy to additional regions to get closer to your users as you see fit.

With all of the infrastructure we deploy, you can get more detailed deployment information on the Deployment Overview page where we represent all of the different Fargate infrastructure we have deployed.

How do I make the API call?

To learn more about triggering an API scan of a file, check out the api-based scanning overview page.

Deployed Regions

Deployed Agents
If there are no currently deployed API agents, then you will only have an Add New Region button available to you. Otherwise, displayed to you are any regions where API agents have been deployed. To setup new regions click the Add New Region button. To edit an existing region's setup, click the specific region pill. Either will populate the bottom portion of the page with the fields that are required to stand up the service.

Default DNS

Default DNS
This will not show on the page as you are setting up a new service region. Once the setup is complete and the load balancer has been created, this page will show the default DNS value. This is the value assigned to the load balancer, not necessarily the value you plan to use with your application teams. This URL is a fully valid and working address, so you can submit API scans to this. It is recommended that you create a CNAME record in your DNS that links a friendlier name to this LB url. This is to ensure validity of the SSL certificate being used. While it still works to simply use the LB url, you would need to disable SSL certificate verification in order to access it.

Network Settings

API Network Settings The Network Settings section defines which network the API Agent will run in and the connectivity to it.

You start by specifying which VPC and Subnets you want the load balancer and Fargate service to run in. Whichever VPC and Subnets you pick ensure that they meet your access needs whether from the outside or inside of your network.

VPC and Subnet Access Importance

You'll notice a Public or Private associated with each VPC. This is an indicator of whether or not the VPC is tied to an Internet Gateway. Thought process being that with an IG in place you will have the required outbound access. The API Agent does not require a public IP address or to be accessible from the public in general, but it does require outbound internet access to get to AWS ECR to pull new Task images.

Console Task Settings - VPCs


You'll notice a Public or Restricted associated with each Subnet. This is an indicator of whether or not the Subnet is outbound routable to the internet. Minimally, the agent task must be able to reach the AWS ECR to pull new Task images and to be able to pull AV signature updates. Two validations are performed for this check. First, we check to see if the NACL associated with the Subnet(s) has outbound open for 0.0.0.0/0. Secondly, we check to see if there is a custom Route Table in place and verify it is routed to an internet gateway.

Console Task Settings - VPCs

Warning

Changing these values will cause the agents to reboot.

You also define who can access the URL via the Inbound Access CIDR. This directly corresponds to an AWS Security Group for the load balancer. Lock this down to the network(s) allowed to access the URL.

The Internet Facing Load Balancer determines whether the load balancer is created with public IPs or not. If you plan to leverage this API endpoint from outside the network, then you'll want this box ticked. If you plan to use the endpoint from resources on the same network then you can uncheck it.

Note

If you decide after the fact you want a different value for Internet Facing Load Balancer you will have to tear the API Agent down and recreate it because there are no AWS APIs to change this value after the fact. You can tear down the API Agent from the Deployment Overview page


The tear down can take upwards of 15 minutes for the load balancer to be removed. Once that has completed you can re-setup the region.

The SSL Certificate ARN is the SSL certificate to associate with the load balancer. You will typically create the certificate to match a friendly URL. If the certificate and API URL do not match, you will get certificate validation warnings and errors. You can code around those, but better to match them up. With the SSL Certificate ARN you have a choice to create your own or leverage ours like the Management Console. The default behavior of the Management Console is to give you persistent access through the cloudstoragesecapp.com domain. We also offer this option for the API Endpoint(s). This allows for a very fast and simple setup for the API Endpoint and doesn't require you to provide your own cert and friendly DNS entry. Your environment could look like the following: console = customerA.cloudstoragesecapp.com and your API endpoint = customerA-api.cloudstoragesecapp.com. And we'll do it all for you.

When first setting up the API Endpoint you'll have an option for New Cert.
API Endpoint New Cert

If you have your own certificate you'd like to use, then simply past the ARN value into this field and ignore the New Cert button. If you'd like us to register your API Endpoint with our DNS as described above, then click the New Cert button to reveal the following popup. Here you can specify the subdomain value you'd like to use for the friendly URL.
API Endpoint New Cert popup
Type in whatever value you'd like to use and check the availability. If it is free, then simply click Create button. We will create the entries in our DNS to make the friendly URL. You may see the following until the certificate and DNS settings are completely setup.
API Endpoint New Cert not ready

Service Settings

API Service Settings

Field                             Description
Min Agents The minimum number of scanning agents you'd like running by default or in the given region. This value cannot be 0.
Max Agents The maximum number of scanning agents you would like to possibly scale to. This number can be anything greater or equal to the minimum.
Scanning Engine                                         The only option at this time is the Sophos engine. Even if you have ClamAV specified for the other types of scanning, the API Agent will still use Sophos.
CPU The amount of vCPU you would like allocated to each agent. Each additional auto-scaled agent would also have this value.
Memory The amount of memory you would like allocated to each agent. Each additional auto-scaled agent would also have this value.
Disk Size (GB) The amount of disk space assigned to each agent. Each additional auto-scaled agent would also have this value.

The default value and included in the AWS pricing is 20GB. If all files you will be scanning are under 15GB in size then keep the default. If there are regions or needs to scan files larger than 15GB in size, then increase this to a size large enough to handle your largest files.

NOTE: There are increased pricing costs for any GB above the 20GB size. Currently, this price per GB per hour is $0.000111, but refer to the AWS Fargate Pricing page to get the latest costs.

Note

The Memory must always be set to a value that is between 2x and 8x of the vCPU.

Examples

    vCPU = 1, then 3gb <= memValue <= 8gb  
    vCPU = 2, then 4gb <= memValue <= 16gb
    vCPU = 3, then 6gb <= memValue <= 24gb
    vCPU = 4, then 8gb <= memValue <= 32gb

Info

The new minimum memory requirement for agents is 3gb. We no longer offer a 2gb memory choice as we saw unstable scanning results with this value.

Warning

Changing these values will cause the agents to reboot.


If the VPC or Subnets are not proper for the agents, you could have trouble with the agents not booting up or entering a constant reboot cycle.

Once you click Setup <region-name> the process of creating the load balancer and API Endpoint agents will commence. This can take a number of minutes to do. While this is happening you will see the following message. You will know it is complete when the yellow bar goes away. Agent Settings

Scaling Considerations

Currently we have seen sufficient performance with scaling out instead of up such that the default values of 1vCPU and 3GB Memory should be all you need for most any workload. Scaling out is driven by two main factors: number of connections to the load balancer and the CPU utilization of the task. Either factor can trigger scaling based on an average over a full minute.

For connections, it is 1000 or greater connections lasting over 1 minute. If 10,000 connections come in all at once and haven't been processed within a minute, then tasks will be scaled out to match the number of connections remaining. If 10,000 were still remaining the scaling policy would start up 9 more tasks.

For CPU utilization, 75% or greater utilization lasting over a minute will trigger scaling. Scaling by CPU utilization behaves slightly differently than connections in that only 1 new instances will be spun up at a time. So if the utilization averages over 75% for a minute then a second instance will spin up. If the utilization of both instances averages over 75% again, then a third instance will spin up and so forth.

Contracting back down after the scaling event takes a bit longer since the load balancer wants to see the connections drained and the cpu utilization stabilized before dropping resources. So you may see tasks linger longer than you'd expect, but they will scale back down.


Last update: September 18, 2021