Architecture Overview
An in-depth overview of our architecture.
Architecture
The architecture seen below supports the object flow as described in the Object Scanning section both in a single region as well as across all regions supported. The Console region will have all components deployed to it. Any additional regions only require the scanning Agent(s) which will report back to the centrally located Console. In addition to this high-level architecture, you can get more details on routing and the public access required on the Deployment Details page.
Architecture - High-level Overview
ECS is the core of our application, which hosts 4 types of services.
The Console service is utilized for configuration of our product and provides the main communication to other AWS services. It also surfaces data from our product for user analysis.
The API Agent provides an endpoint where files can be sent and scanned before entering a storage volume.
The Event Agent scans files as they land into an S3 bucket by leveraging SNS and SQS.
The Scheduled & On-demand Agent scans pre-existing objects in a storage volume using Fargate Run Tasks.
Users and apps can log into our management console service through Cognito, or send files directly to our API scanning agent through a load balancer.
Objects can be uploaded either directly into the supported storage volumes or uploaded to S3 via our API Agent.
In AWS, we currently scan S3, EBS, EFS, and FSx.
Integrations are handled through our Console service.
The Console stores and reads configuration details from DynamoDB and Parameter Store, for example to ensure consistency when scaling agents or restarting tasks.
The Console logs all application activity in CloudWatch, which can be integrated with most SIEM tools.
ECS is the core of our application, which hosts 4 types of services.
The Console service is utilized for configuration of our product and provides the main communication to other AWS services. It also surfaces data from our product for user analysis.
The API Agent provides an endpoint where files can be sent and scanned before entering a storage volume.
The Event Agent scans files as they land into an S3 bucket by leveraging SNS and SQS.
The Scheduled & On-demand Agent scans pre-existing objects in a storage volume using Fargate Run Tasks.
Users and apps can log into our management console service through Cognito, or send files directly to our API scanning agent through a load balancer.
Objects can be uploaded either directly into the supported storage volumes or uploaded to S3 via our API Agent.
In AWS, we currently scan S3, EBS, EFS, and FSx.
Integrations are handled through our Console service.
The Console stores and reads configuration details from DynamoDB and Parameter Store, for example to ensure consistency when scaling agents or restarting tasks.
The Console logs all application activity in CloudWatch, which can be integrated with most SIEM tools.
To scan objects in regions separate from the Intial Deployment Region, we spin up Scanning Agents in that region to perform scans so the files never leave the region. Results are then sent back to the Console service.
All application information that takes place in secondary deployment regions are logged into that region's CloudWatch log groups.
ECS is the core of our application, which hosts 4 types of services.
The Console service is utilized for configuration of our product and provides the main communication to other AWS services. It also surfaces data from our product for user analysis.
The API Agent provides an endpoint where files can be sent and scanned before entering a storage volume.
The Event Agent scans files as they land into an S3 bucket by leveraging SNS and SQS.
The Scheduled & On-demand Agent scans pre-existing objects in a storage volume using Fargate Run Tasks.
Users and apps can log into our management console service through Cognito, or send files directly to our API scanning agent through a load balancer.
Objects can be uploaded either directly into the supported storage volumes or uploaded to S3 via our API Agent.
In AWS, we currently scan S3, EBS, EFS, and FSx.
Integrations are handled through our Console service.
The Console stores and reads configuration details from DynamoDB and Parameter Store, for example to ensure consistency when scaling agents or restarting tasks.
The Console logs all application activity in CloudWatch, which can be integrated with most SIEM tools.
To scan objects in regions separate from the Intial Deployment Region, we spin up Scanning Agents in that region to perform scans so the files never leave the region. Results are then sent back to the Console service.
All application information that takes place in secondary deployment regions are logged into that region's CloudWatch log groups.
For multiple accounts, the Console deploys a Cross Account Role with permissions so that CSS services can access and scan storage volumes from separate accounts. Note that scanning infrastructure remains in the primary AWS account where your Console was deployed. Similar to a multi-region deployment, CSS will spin up Scanning Agents in each region where data is scanned.
When linking an account in the CSS console, we deploy resources in the new CSS Resource Group created in Azure. Those resources will have access to the Blob Containers that reside within the linked Azure Account.
Once the Azure Account is linked, Azure Blob Containers will be made available to be scan through the CSS Console in the Protection > Azure Blob Containers page. You can select blob containers to scan or run scans on a schedule basis.
We offer two types of scanning for Azure: Event-Scanning and Retro Scanning.
Event-based scanning scans files as they drop into the Blob Container.
Users upload data to storage containers.
The storage account's system topic has an Event Grid Subscription tied to it that points to Azure Queue Storage as its event handler.
The Azure Queue Storage receives information about the object via the Event Grid Subscription.
The Container Application pulls information from the queue and scans the object that the notification refers to.
Results are tagged onto the object.
Results and metering are sent back to the CSS Console.
Retro-based scanning scans the entire Blob Container.
Users upload data to storage containers.
The Container Application runs a Crawl Job that evaluates the items to be scanned.
The Crawl Job places objects in Azure Queue Storage.
The Container Application pulls information from the queue and runs a Scan Job to scan the objects in the Queue.
Results and metering are sent back to the CSS Console.
Excluding the Event Grid Subscription, note that all of these resources and cross-permissions are only deployed inside of the CSS Resource Group through the Azure Resource Manager. The rest of the organizations and projects inside of your Azure account remain untouched.
When linking an account in the CSS console, we deploy resources in a new CSS Project created in GCP through our terraform module. Those resources will have access to the customer project(s) denoted in the deployments parameters (projects_to_protect).
Once the projects are linked, GCP buckets will be made available to be scan through the CSS Console in the Protection > GCP Buckets page. You can select buckets to scan or run scans on a schedule basis.
We offer Retro scanning for GCP.
Users or apps place files into Google Cloud Storage.
The Console Service initiates a scan request to the Cloud Run service that resides in CSS' Project that is provisioned via Terraform.
The Cloud Run Job accesses the customer's Cloud Storage and loads a list of its objects in memory, procedurally scanning each item.
Results are returned to the Console service for processing.
(Optional) The Cloud Run Job will move infected files to a Quarantine Cloud Storage that resides in CSS' Project.
All scan results are logged in AWS CloudWatch logs Agent.ScanResults, Problem Files are reported in the console, and objects in Google Cloud Storage are tagged with the scan result as well via the file metadata.
Note that all of these resources and cross-permissions are only deployed inside of the CSS Project through Terraform. The rest of the organizations and projects inside of your GCP account remain untouched.
Platform Services
While many services are used (ECS Fargate, App Config, CloudWatch, CloudFormation, DynamoDB, SNS, SQS, IAM) to deliver the Antivirus for Amazon S3 solution, two will be called out here. CloudWatch and IAM are leveraged for logging and permissions respectively. These are the usual questions we get from customers:
How do I check the logs?
What are you doing behind the scenes (permissions wise)?
We wanted to make sure you had those bases covered with the information below.
CloudWatch Log Group Overview
Log groups for the Console
Log groups for the Agent
ScanResults
Scan results for clean, infected, error, or unscannable files.
Infected:
2020-08-24T15:15:33.067-06:00 2020-08-24 21:15:33.0672|INFO|InfectedScanResults|{"guid":"e132dc70-4582-476a-bb52-c57425c9792e","dateScanned":"2020-08-24T21:15:32.7952943Z","bucketName":"demo-destination-bucket","key":"virus/7hXNy9okVjpszoFP_virus_388_eicarcom2.zip","scanResult":"Infected","actionTaken":"Move","detectedVirus":"Win.Test.EICAR_HDB-1","virusUploadedBy":"AWS:AROA3K5IVNMVEDVQSN5PM:demo-bucket-transfer","errorMessage":"","fileExists":true,"movedTo":"cloudstoragesecquarantine-y6uajej-7xxxxxxxxxxx8-us-east-1","region":"us-east-1","accountId":"7xxxxxxxxxxx8"}
2020-08-24T15:15:33.067-06:00 2020-08-24 21:15:33.0672|INFO|InfectedScanResults|
{
"guid": "e132dc70-4582-476a-bb52-c57425c9792e",
"dateScanned": "2020-08-24T21:15:32.7952943Z",
"bucketName": "demo-destination-bucket",
"key": "virus/7hXNy9okVjpszoFP_virus_388_eicarcom2.zip",
"scanResult": "Infected",
"actionTaken": "Move",
"detectedVirus": "Win.Test.EICAR_HDB-1",
"virusUploadedBy": "AWS:AROA3K5IVNMVEDVQSN5PM:demo-bucket-transfer",
"errorMessage": "",
"fileExists": true,
"movedTo": "cloudstoragesecquarantine-y6uajej-7xxxxxxxxxxx8-us-east-1",
"region": "us-east-1",
"accountId": "7xxxxxxxxxxx8"
}Clean:
2020-08-24T15:15:33.243-06:00 2020-08-24 21:15:33.2432|INFO|CleanScanResults|{"guid":"5cab2514-5982-4323-bdbc-77540dca973d","dateScanned":"2020-08-24T21:15:33.186175Z","bucketName":"demo-destination-bucket","key":"1mb/xglRNavTNgA67qim_temp_1mb_file94857.txt","scanResult":"Clean","actionTaken":"None","detectedVirus":"","virusUploadedBy":"","errorMessage":"","fileExists":true,"movedTo":"","region":"us-east-1","accountId":"7xxxxxxxxxxx8"}
2020-08-24T15:15:33.344-06:00 2020-08-24 21:15:33.3444|INFO|CleanScanResults|
{
"guid": "b589b129-ac54-493c-886c-30016899f3b9",
"dateScanned": "2020-08-24T21:15:33.2737108Z",
"bucketName": "demo-destination-bucket",
"key": "1mb/xRP72vFa1Ays2Qr9_temp_1mb_file94075.txt",
"scanResult": "Clean",
"actionTaken": "None",
"detectedVirus": "",
"virusUploadedBy": "",
"errorMessage": "",
"fileExists": true,
"movedTo": "",
"region": "us-east-1",
"accountId": "7xxxxxxxxxxx8"
}Error:
2020-08-24T15:15:00.132-06:00 2020-08-24 21:15:00.1314|INFO|ErrorScanResults|{"guid":"5806ced2-688a-45d0-a2cb-71717176e66e","dateScanned":"2020-08-24T21:14:59.6058615Z","bucketName":"webinar-other-account-bucket-2","key":"ConsoleCloudFormationTemplate.yaml","scanResult":"Error","actionTaken":"None","detectedVirus":"","virusUploadedBy":"","errorMessage":"Unable to access the remote account.","fileExists":true,"movedTo":"","region":"us-east-1","accountId":"7xxxxxxxxxxx7"}
2020-08-24T15:15:00.206-06:00 2020-08-24 21:15:00.2055|INFO|ErrorScanResults|
{
"guid": "c95dfbb1-2853-49e1-ace9-c2ae05bbf32a",
"dateScanned": "2020-08-24T21:14:59.6058615Z",
"bucketName": "webinar-other-account-bucket-2",
"key": "ConsoleCloudFormationTemplate.yaml",
"scanResult": "Error",
"actionTaken": "None",
"detectedVirus": "",
"virusUploadedBy": "",
"errorMessage": "Unable to access the remote account.",
"fileExists": true,
"movedTo": "",
"region": "us-east-1",
"accountId": "7xxxxxxxxxx7"
}Log groups for ECS
As of version 6.06 we enable ECS logging by default. These logs will be shown in the following log groups.
For each of these log groups you will see your seven character application ID in the title of each log group as noted below by the AppID between the ECS and type of ECS service the log is for.
IAM Permissions Review
We have been able to simplify the management and delivery of the solution such that there are very few tasks the administrator is required to perform inside the AWS Console. As a result, the Console and EventAgent have a number of permissions assigned to them within their respective roles to allow them to perform the actions needed on your behalf. In all cases, we went with a least privilege model wherever possible. There are a few instances where we have assigned * when it is required. Below you will find a review of the two IAM Roles we create and assign to the Console and scanning Agents.
Please review and Contact Us if you have any questions we can clear up for you.
The permission descriptions below follow the format:
- system-name
- permission 1
- reason it is needed
- ...
- reason it is needed
- permission n
- reason it is neededPermissions Policies
Console Role
Console Role Customer Inline Policies
Console Role Customer Managed Policies
Scan Engines
Antivirus for Amazon S3 has been built in such a way that the underlying scanning engine can be exchanged with other scanning engines as needed or desired. There are three engines included out of the box:
Sophos - a well known enterprise solution that offers speed, great accuracy and large file scanning
ClamAV - a widely used open source antivirus engine for detecting trojans, viruses, malware & other malicious threats
CSS Premium - our latest scanning engine that can be used as either a primary scanning engine or as a secondary scanning engine to increase the efficacy of your scan
Agent checks every 15 minutes Vendor typically updates 4 times per day
195GB 5TB with extra large file scanning File Types
Faster engine; refer to Sizing Discussion
Signature Based
Yes
Agent checks every hour Vendor typically updates once per day
2GB File Types
Good performance engine; refer to Sizing Discussion
Signature Based
Yes
Agent checks every 15 minutes Typically updates one to four times per day
195GB through an ECS task 5TB with extra large file scanning File Types
Great performance; data to be released in the near future
Signature Based
Yes
Antivirus for Amazon S3 has the ability to use multiple scanning engines configured serially to ensure the highest level of efficacy and protection. Antivirus for Amazon S3 updates virus definitions as defined above as well as with each reboot / new spin up.
Last updated