Cloud Storage Security Help Docs
Release Notes
  • Introduction
  • Getting Started
    • How to Subscribe
      • Pay-As-You-Go (PAYG)
      • Bring Your Own License/GovCloud (BYOL)
      • AWS Transfer Family
    • How to Deploy
      • Steps to Deploy
      • Advanced Deployment Considerations
      • AWS Transfer Family
    • How to Configure
  • Console Overview
    • Dashboard
    • Malware Scanning
      • AWS
        • Buckets
        • Amazon EBS Volumes
        • Amazon EFS Volumes
        • Amazon FSx Volumes
        • WorkDocs Connections
      • Azure
        • Blob Containers
      • GCP
        • GCP Buckets
    • See What's Infected
      • Findings
      • Malware History
      • Results
    • Schedules
    • Monitoring
      • Error Logs
      • Bucket Settings
      • Deployment
      • Jobs
      • Notifications
      • Storage Assessment
      • Usage
    • Configuration
      • Classification Rule Sets
      • Classification Custom Rules
      • Scan Settings
      • Console Settings
      • AWS Integrations
      • Job Networking
      • API Agent Settings
      • Proactive Notifications
      • License Management
      • Event Agent Settings
    • Access Management
      • Manage Users
      • Manage Accounts
        • Linking an AWS Account
        • Linking an Azure Account
        • Linking a GCP Account
      • Manage Groups
    • Support
      • Getting Started
      • Stay Connected
      • Contact Us
      • Documentation
  • Product Updates
  • How It Works
    • Scanning Overview
      • Event Driven Scanning for New Files
      • Retro Scanning for Pre-Existing Files
      • API Driven Scanning
    • Architecture Overview
    • Deployment Details
    • Sizing Discussion
    • Integrations
      • AWS Security Hub
      • AWS CloudTrail Lake
      • AWS Transfer Family
      • Amazon GuardDuty
      • Amazon Bedrock
    • Demo Videos
    • Scanning APIs
    • SSO Integrations
      • Entra ID SSO Integration
      • Okta SSO Integration
  • Frequently Asked Questions
    • Getting Started
    • Product Functionality
    • Architecture Related
    • Supported File Types
  • Troubleshooting
    • CloudFormation Stack failures
    • Cross-Region Scanning on with private network
    • API Scanning: Could not connect to SSL/TLS (v7)
    • Password not received after deployment
    • Conflicted buckets
    • Modifying scaling info post-deployment
    • Objects show unscannable with access denied
    • Remote account objects not scanning
    • My scanning agents keep starting up and immediately shutting down
    • I cannot access the management console
    • Linked Account Out of Date
    • Rebooting the Management Console
    • Error when upgrading to the latest major version
    • I Cannot Create/Delete an API Agent
  • Release Notes
    • Latest (v8)
    • v7
    • v6 and older
  • Contact Us & Support
  • Data Processing Agreement
  • Privacy Policy
Powered by GitBook
On this page
  • Do I need to make any changes to my application to use the product?
  • Can I scan S3 objects from more than one account from within the same deployment?
  • Can I setup a staging bucket for all of my files to first land in and then move them to a production bucket after they have been found to be clean?
  • What ports do I need open for the product to function properly?
  • Can I change the CIDR range, VPC or Subnets post deployment for the console and agents?
  • Do you use AWS Lambdas or EC2 Instances?
  • Do you support AWS Control Tower or Landing Zone?
  • Can I leverage Single Sign On (SSO) with your product?
  • Additional Actions Required
  • How can we scan your console/agent images for vulnerabilities?
  • How do I completely tear down CSS Infrastructure?
  1. Frequently Asked Questions

Architecture Related

Below are some of the most common questions related to Architecture

PreviousProduct FunctionalityNextSupported File Types

Last updated 4 months ago

Do I need to make any changes to my application to use the product?

No. The Antivirus for Amazon S3 solution will fit into your existing workflow. You do not have to make any changes to your current workflow.

Can I scan S3 objects from more than one account from within the same deployment?

Yes, Antivirus for Amazon S3 supports cross-account scanning. This means you can centrally install the console and scanning agents to protect not only the account you are deployed within, but also any other AWS account where you can install a cross-account role.

Check out the documentation for more details.

Can I setup a staging bucket for all of my files to first land in and then move them to a production bucket after they have been found to be clean?

Two Bucket System

Yes, it is very easy to setup a two bucket system and we have many customers using this approach. We provide two methods of achieving this.

First, within the Configuration > Scan Settings menu there is a configuration option for . This allows for a quick and easy way to choose your source region or bucket(s) and a destination bucket to promote your clean files. With this option, our agent will handle promoting the clean files as part of the scanning process.

Alternatively, you can also implement this approach with a combination of and , as described below.

  1. Create staging bucket or utilize existing bucket

  2. Turn from the Antivirus for Amazon S3 console

  3. Create a Python (latest version) Lambda Function with sample code provided below

  4. Make adjustments to the Lambda settings with the information below

  5. Subscribe Lambda to SNS Notifications Topic

  6. Add IAM permissions to Lambda with provided permission blocks below

  7. Modify Topic Subscription to filter down to clean objects with provided filter block below

  8. Test clean and "not clean" files to ensure behavior is as expected

*LinkedAccountBuckets: Optionally you can make your two-bucket-system on linked accounts buckets as well, adding this pieces. **Multi-partFiles Lambda: This lambda also involves step functions to handle multi-part files and large files

Sample Copy Lambda

The code below is a starting point and does work out of the box, but more can be done with it and to it. Feel free to do so.

import json
import boto3
import os
from botocore.exceptions import ClientError, ParamValidationError
import random
from urllib import parse

def lambda_handler(event, context):

    try:
        print(json.dumps(event))

        SOURCE_BUCKET = os.getenv("SOURCE_BUCKET", 'any')
        DESTINATION_BUCKET = os.getenv("DESTINATION_BUCKET", '<some failover bucket>')
        DELETE_STAGING = os.getenv("DELETE_STAGING", 'no')

        #print("Source bucket is:" + SOURCE_BUCKET)
        #print("Destination bucket is:" + DESTINATION_BUCKET)

        record = event['Records'][0]
        messageBucket = record['Sns']['MessageAttributes']['bucket']

        print("The messageBucket value is:")
        print(messageBucket['Value'])

        if (messageBucket['Value'] == SOURCE_BUCKET or SOURCE_BUCKET == 'any'):

            message = json.loads(record['Sns']['Message'])

            #print("The message content is:" + str(message))
            #print("The message key is: " + message['key'])

            s3 = boto3.resource('s3')

            copy_source = {
                'Bucket': messageBucket['Value'],
                'Key': message['key']
                }

            if 'PartsCount' in s3.meta.client.head_object(Bucket=messageBucket['Value'], Key=message['key'], PartNumber=1):
                # get the tags, then copy with tags specified
                #print("doing a multipart copy with tags")
                try:

                    tagging = s3.meta.client.get_object_tagging(Bucket=messageBucket['Value'], Key=message['key'])
                    #print("get object tagging = " + str(tagging))

                    s3.meta.client.copy(copy_source, DESTINATION_BUCKET, message['key'], ExtraArgs={'Tagging': parse.urlencode({tag['Key']: tag['Value'] for tag in tagging['TagSet']})})

                except Exception as e:
                    print(e)
                    raise(e)
            else:
                # copy as normal
                #print("doing a normal copy")
                try: 
                    s3.meta.client.copy_object(CopySource=copy_source, Bucket=DESTINATION_BUCKET, Key=message['key'])

                except Exception as e:
                    print(e)
                    raise(e)



            print("Copied:  " + message['key'] + "  to production bucket:  " + DESTINATION_BUCKET)

            #print("Delete files: " + DELETE_STAGING)

            if (DELETE_STAGING == 'yes'):
                try:

                    s3.meta.client.delete_object(Bucket=SOURCE_BUCKET, Key=message['key'])
                    print("Deleted:  " + message['key'] + "  from source bucket:  " + SOURCE_BUCKET)

                except Exception as e:
                    print(e)
                    raise(e)

            return {
            'statusCode': 200,
            'body': json.dumps('Non-infected object moved to production bucket')
            }   


        return {
                'statusCode': 200,
                'body': json.dumps('Not from the Staging bucket')
            }

    except ClientError as e:
        return {
            'statusCode': 400,
            'body': "Unexpected error: %s" % e}
    except ParamValidationError as e:
        return {
            'statusCode': 400,
            'body': "Parameter validation error: %s" % e}

Make Adjustments to Lambda Settings

There are some adjustments to the Lambda you'll probably need to make:

  • Set the environment variables for the Staging Bucket and Production Bucket names as seen here:

Field
Description

DELETE_STAGING

yes or no value based on whether you want the original file to be deleted after the copy has occurred

DESTINATION_BUCKET

Clean or Production bucket name identifying where to copy clean files to

SOURCE_BUCKET

Dirty or Staging bucket name identifying the originating bucket Note: you can leverage SNS Topic Filtering to eliminate the need for this value and the if check inside the code

  • Change the Time Out under General Configuration to a value that will work for the typical file sizes you deal with. The larger the file size, the longer you may want to make it so the lambda doesn't time out before the copy finishes.

Permissions to add to Lambda Role

Add an Inline Policy to the Lambda Role that was created when you created the Lambda. Paste the below into the JSON screen and change the sections that have <> to match your staging bucket name and production destination buckets

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObjectTagging",
                "s3:GetObject",
                "s3:GetObjectTagging",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<staging bucket name>",
                "arn:aws:s3:::<staging bucket name>/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:PutObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<production destination bucket name>/*",
                "arn:aws:s3:::<production destination bucket name>"
            ]
        }
    ]
}

Subscription Filter for Clean Results

Go to the SNS Topic itself and find the subscription created for the Lambda. Edit the filter settings by pasting the below value in. You can modify the filter further to include more scan results and even filter down by bucket. Bucket filtering is a good idea if you have event based scanning setup for any other bucket and you do not want the lambda to copy those clean files over as well.

{
    "notificationType": [
        "scanResult"
    ],
    "scanResult": [
        "Clean"
    ]
}

Alternatively, with bucket filtering:

{
    "notificationType": [
        "scanResult"
    ],
    "scanResult": [
        "Clean"
    ],
    "bucket": [
        "<staging bucket name>"
    ]
}

Permissions on linked account buckets:

  • Staging bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "<lambda-role-arn>"
            },
            "Action": [
                "s3:DeleteObjectTagging",
                "s3:GetObject",
                "s3:GetObjectTagging",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<staging-bucket>",
                "arn:aws:s3:::<staging-bucket>/*"
            ]
        }
    ]
}
  • Production bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "<lambda-role-arn>"
            },
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:PutObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<production-bucket>/*",
                "arn:aws:s3:::<production-bucket>"
            ]
        }
    ]
}

Full Lambda for multi-part files handling

import json
import boto3
from botocore.exceptions import ClientError, ParamValidationError
from urllib import parse
import time
import math

def lambda_handler(event, context):

    try:
        print(json.dumps(event))

        record = event['Records'][0]
        message = json.loads(record['Sns']['Message'])

        # Modify destination Bucket as appropriate
        sourceBucket = str(message['bucketName'])
        destinationBucket = sourceBucket

        # Modify destination key as appropriate
        sourceKey = str(message['key'])
        destinationKey = sourceKey.replace("incoming", "landing", 1)

        # Set to appropriate step function state machine ARN
        stateMachineArn = 'arn:aws:states:<region>:<account_id>:stateMachine:<stateMachineName>'

        print("The source bucket is:  " + sourceBucket)
        print("The destination bucket is:  " + destinationBucket)
        print("The source key is:  " + sourceKey)
        print("The destination key: " + destinationKey)
        print("The State Machine ARN: " + stateMachineArn)

        s3 = boto3.resource('s3')

        copy_source = {
            'Bucket': sourceBucket,
            'Key': sourceKey
        }

        meta_data = s3.meta.client.head_object(
            Bucket=sourceBucket, Key=sourceKey)
        parts_meta = s3.meta.client.head_object(
            Bucket=sourceBucket, Key=sourceKey, PartNumber=1)

        partsCount = parts_meta["PartsCount"] if 'PartsCount' in parts_meta else 0

        if partsCount > 0:
            totalContentLength = meta_data["ContentLength"]
            partSize = math.floor(totalContentLength / partsCount)

            print(f'FileSize: {totalContentLength} -- PartSize: {partSize} -- PartsCount: {partsCount}')

            tagging = s3.meta.client.get_object_tagging(
                Bucket=sourceBucket, Key=sourceKey)
            tagging_encoded = parse.urlencode(
                {tag['Key']: tag['Value'] for tag in tagging['TagSet']})

            if totalContentLength >= 25_000_000_000:
                source_bucket_and_key = f'{sourceBucket}/{sourceKey}'

                try:
                    # 25GB or larger file, use step function to copy due to lambda run time limit
                    print("Using step function to copy extra large file")

                    # Initiate multi-part upload
                    kwargs = dict(
                        Bucket=destinationBucket,
                        Key=destinationKey,
                        Tagging=tagging_encoded,
                        Metadata=meta_data['Metadata'],
                        StorageClass=meta_data['StorageClass'] if 'StorageClass' in meta_data else None,
                        ServerSideEncryption=meta_data['ServerSideEncryption'] if 'ServerSideEncryption' in meta_data else None,
                        SSEKMSKeyId=meta_data['SSEKMSKeyId'] if 'SSEKMSKeyId' in meta_data else None,
                        BucketKeyEnabled=meta_data['BucketKeyEnabled'] if 'BucketKeyEnabled' in meta_data else None,
                        ObjectLockMode=meta_data['ObjectLockMode'] if 'ObjectLockLegalHoldStatus' in meta_data and 'ObjectLockRetainUntilDate' in meta_data else None,
                        ObjectLockRetainUntilDate=meta_data['ObjectLockRetainUntilDate'] if 'ObjectLockLegalHoldStatus' in meta_data and 'ObjectLockRetainUntilDate' in meta_data else None,
                        ObjectLockLegalHoldStatus=meta_data[
                            'ObjectLockLegalHoldStatus'] if 'ObjectLockLegalHoldStatus' in meta_data else None
                    )

                    multipart_upload_response = s3.meta.client.create_multipart_upload(
                        **{k: v for k, v in kwargs.items() if v is not None}
                    )

                    uploadId = multipart_upload_response['UploadId']

                    # kick off step function executions
                    sfn_client = boto3.client('stepfunctions')

                    parts = []
                    baseInput = {
                        'Bucket': destinationBucket,
                        'Key': destinationKey,
                        'CopySource': source_bucket_and_key,
                        'SourceBucket': sourceBucket,
                        'SourceKey': sourceKey,
                        'UploadId': uploadId,
                        'PartsCount': partsCount,
                        'CompleteUpload': False
                    }

                    for partNumber in range(1, partsCount + 1):
                        byteRangeStart = (partNumber - 1) * partSize
                        byteRangeEnd = byteRangeStart + partSize - 1

                        if byteRangeEnd > totalContentLength or partNumber == partsCount:
                            byteRangeEnd = totalContentLength - 1

                        parts.append({
                            'PartNumber': partNumber,
                            'CopySourceRange': f'bytes={byteRangeStart}-{byteRangeEnd}'
                        })

                        if len(parts) == 250 and partNumber != partsCount:
                            executionInput = dict(baseInput)
                            executionInput['Parts'] = parts

                            sfn_client.start_execution(
                                stateMachineArn=stateMachineArn,
                                name=f'copy_parts_{time.time() * 1000}',
                                input=json.dumps(executionInput)
                            )

                            parts.clear()

                    executionInput = dict(baseInput)
                    executionInput['Parts'] = parts
                    executionInput['CompleteUpload'] = True

                    sfn_client.start_execution(
                        stateMachineArn=stateMachineArn,
                        name=f'copy_parts_{time.time() * 1000}',
                        input=json.dumps(executionInput)
                    )

                    print("Step function execution started.")

                    return {
                        'statusCode': 200,
                        'body': json.dumps('Non-infected object moved to production bucket')
                    }
                except Exception as e:
                    print(
                        f'Failed to execute step function to copy {source_bucket_and_key}')
                    print(e)
                    raise (e)
            else:
                # get the tags, then copy with tags specified
                print("Performing a multipart copy with tags")
                try:
                    s3.meta.client.copy(copy_source, destinationBucket, destinationKey, ExtraArgs={
                                        'Tagging': tagging_encoded})

                except Exception as e:
                    print(e)
                    raise (e)
        else:
            # copy as normal
            print("Performing a normal copy")
            try:
                s3.meta.client.copy_object(
                    CopySource=copy_source, Bucket=destinationBucket, Key=destinationKey)

            except Exception as e:
                print(e)
                raise (e)

        print(
            f'Copied: {source_bucket_and_key} to destination: {destinationBucket}/{destinationKey}')

        try:
            s3.meta.client.delete_object(Bucket=sourceBucket, Key=sourceKey)
            print(f'Deleted: {source_bucket_and_key}')

        except Exception as e:
            print(e)
            raise (e)

        return {
            'statusCode': 200,
            'body': json.dumps('Non-infected object moved to production bucket')
        }

    except ClientError as e:
        return {
            'statusCode': 400,
            'body': "Unexpected error: %s" % e}
    except ParamValidationError as e:
        return {
            'statusCode': 400,
            'body': "Parameter validation error: %s" % e}

Custom/own Quarantine Bucket

If you decide to use your own quarantine bucket, you can use these same steps for a 2-bucket-system. You only need to go to Configuration > Scan Settings and change the action for infected files to Keep and change the "Clean" subscription on step 7 for "Infected"

How the 2 Bucket System Flows:

What ports do I need open for the product to function properly?

Port 443 for:

  • Outbound for Lambda calls

  • Outbound Console and Agent access to Elastic Container Repository (ECR)

  • Inbound access to Console for public access

    • Public access is not required as long as you have access via private IP

Port 80 for:

  • ClamAV signature updates

Can I change the CIDR range, VPC or Subnets post deployment for the console and agents?

Do you use AWS Lambdas or EC2 Instances?

Neither. Antivirus for Amazon S3 infrastructure is built around AWS Fargate containers. We wanted to be serverless like Lambda and faster and more flexible than EC2s. Fargate containers give you persistence and other benefits that Lambdas aren't prepared to give you yet. We explored Lambda and do see some advantages there, but not enough to win out over AWS Fargate containers.

Do you support AWS Control Tower or Landing Zone?

A landing zone is a well-architected, multi-account AWS environment that's based on security and compliance best practices. AWS Control Tower automates the setup of a new landing zone using best-practices blueprints for identity, federated access, and account structure.

Can I leverage Single Sign On (SSO) with your product?

Yes you can leverage SSO with our solution. Antivirus for Amazon S3 utilizes Amazon Cognito for user management. Amazon Cognito allows SAML integrations. Leveraging this capability we can utilize various providers as part of SSO into our solution, both from the SSO Dashboard as well as from within the application itself.

We've documented SSO integration for Entra ID and Okta below:

Examples below:

Okta

Entra ID

Additional Actions Required

Okta identities will be auto-created within Amazon Cognito (and therefore Antivirus for Amazon S3) as simple Users and not Admins. They are also not assigned to a particular Group. This state enables them to login, but manage nothing. One-time only you will need to assign the user to a group and the admin role. After this initial assignment, each subsequent login will allow for proper management.

SSO users will also standout as their username will be created from the SSO sign in process.

How can we scan your console/agent images for vulnerabilities?

You'll need to install and use the AWS CLI to authenticate to Amazon Elastic Container Registry and download the container images using the commands.

Below is a sample of the commands to pull our images from ECR for v7.01.001. However, you'll need to navigate to the Launch this software page on our AWS Marketplace listing to get the commands for our latest release.

aws ecr get-login-password \
    --region us-east-1 | docker login \
    --username AWS \
    --password-stdin 564477214187.dkr.ecr.us-east-1.amazonaws.com
    
CONTAINER_IMAGES="564477214187.dkr.ecr.us-east-1.amazonaws.com/cloud-storage-security/console:v7.01.001,564477214187.dkr.ecr.us-east-1.amazonaws.com/cloud-storage-security/agent:v7.01.001"    

for i in $(echo $CONTAINER_IMAGES | sed "s/,/ /g"); do docker pull $i; done

How do I completely tear down CSS Infrastructure?

If using CloudFormation, the action of just deleting the stack may miss additional infrastructure created throughout the course of using our product. We recommend that you go to Monitoring > Deployment and Delete Application so we can clean up infrastructure for you before you delete the stack.

If you need any help getting this setup, please as we are happy to help.

You can now setup rather than reach out over the internet. This will allow you to setup an Amazon S3 bucket for the solution to look at.

You can get a more detailed view and additional options for routing on the . In either the standard deployment or the VPC Endpoints deployment, with local signature updates you can remove all non-AWS calls from the application run space. With VPC Endpoint you can remove almost all public calls as well.

Yes. The page gives you the option to modify the inbound Security Group rules, the VPC and Subnets and the specs of the task (vCPU and Memory). The page allows you to change the VPC and Subnets the agents run in, the specs of the task (vCPU and Memory) as well as all the scaling configuration aspects.

We do leverage two lambdas for the , but not for any of the workload at this time. If you are interested in a lambda-driven solution, please to let us know. We are always exploring the best way to build and run our solution.

Antivirus for Amazon S3 is now tightly integrated with , and is designed to work within the landing zone context. Antivirus for Amazon S3 can be centrally deployed in a Security Services account while leveraging to scan all other accounts. You can learn more about here.

For

For

Upon customer request, we have documented the steps to get GSuite working as your SSO provider. Please leverage the document below. We have had other customers leverage these steps (along with the Okta and Entra ID write up) to setup other providers as well such as Keycloak.

You can access our images by either or using the Container images commands on the Launch this software page on our AWS Marketplace listing.

More details are located at .

Contact Us
Deployment Details page
Console Settings
Agent Settings
AWS Control Tower
Linked Accounts
AWS Control Tower
Okta
GSuite Setup Instructions
this page
Linked Accounts
AV Two-Bucket System Configuratio
n
Event Based scanning
Proactive Notifications
Contact Us
local signature updates
bucket protection on for this bucket
downloading them locally to your ECR
subdomain registration
Sign In lookslike
Okta UI
portal.azure.com > Enterprise Applications
Entra ID