Skip to content

Storage Assessment


The Storage Assessment page gives you detailed information about your Amazon S3 storage estate. Included are file level details and data trends that are useful for understanding you Amazon S3 environment. Included on the page are widgets that give you an overall feel, relevant charts showing an S3 Overview, File Information, Bucket Information and Trends. You can filter the information shown by bucket, region, account, and/or date. Also you can see the date of the last scan under the Storage Assessment title. You can also see the date your information is being displayed for. And finally under the Filters you can see the time frame you are viewing data for.

storage assessment page

Top Row Informationals

The top row of widgets gives you a quick view to specific details around your S3 environment. The top row gives you the aggregate totals of the latest snapshot for Total Data in your S3 environment, Total Objects in your S3 environment, Buckets and Regions enabled with Storage Assessment, Total Objects Scanned with our application, and Total Objects Encrypted in your S3 environment.

storage assessment widgets

Assessment Visuals

Disclaimer

Any chart labeled Top X will show 10 if there are more than 10 total buckets in your S3 account. Also the first 2 panels charts will only show you information about the latest snapshot. For example if you have a 30 day time frame selected it will only display information about the latest day (yesterday) as that contains the most up to date information.

S3 Overview

This panel contains 4 charts about your S3 environment. Going clockwise from the top left: Total Object Count and Size chart of your current S3 environment, Top 10 Largest Buckets in GB's, Percent Scanned of 10 Largest Buckets, and Percent Encrypted of 10 Largest Buckets.

storage assessment s3 overview

File Information

The panel contains 3 charts about files in your S3 environment. Going clockwise from the top left: Top 10 File Types by size, Top 10 File Types by count, and File Ages of all files in your S3 environment.

storage assessment s3 fileinformation

Bucket Information

This panel contains 4 charts about Buckets in your S3 environment. Going clockwise from the top left: Public and Private Bucket Count by Region Encrypted and Unencrypted Bucket Count by Region, Percentage of a regions scanned object, and Percentage of a regions encrypted objects. The top 2 charts information we get from our Bucket information collection and the bottom 2 charts information we get from Storage Assessment data. If you do not have Percent Scanned enabled that graph will display 0's for all regions

storage assessment s3 bucketinformation

The trends tab contains 3 graphs that will only be visible if your time frame is larger than 1 day. They will display trends for whatever time frame is selected. First Graph is Bucket Object count by day, then Bucket Size by day, then Percent Scanned of each bucket by day.

storage assessment trends


Keywords

Storage Assessment

An assessment of aggregated stats by our application using a manual bucket crawl initially and then AWS’s Inventory configuration report for the remaining time.

Manual Crawl

Our application manually crawls your buckets and gathers information about the files sizes, last modified date, encryption status, and file type.

Inventory Configuration

A snap shot report generated once a day by AWS that contains information about your bucket contents. It contains file name, size, last modified date, encryption status, storage class, intelligent tiering access tier, and the name the bucket it resides in. Inventory configuration can have 2 status: Enabled and Disabled. Enabled means that a report is generated and delivered every night. Disabled means that Inventory Configuration settings are primed on the bucket but no report is generated and delivered. Our application configures the Inventory Config to deliver the reports to a directory in your quarantine bucket in the same buckets region.

Percent Scanned

This refers to the percent of objects scanned in your bucket. It takes the key name from the Inventory Config and checks the file for tags in S3 that match our applications tags. We then take the count of objects with matching tags and find the percentage against your buckets total object count.

Disclaimer

Since the Inventory Config’s report is generated separately from when we aggregate it there could be a discrepancy if files are deleted during that time frame that are in the report.

Percent Encrypted

We can get the file encryption status information from the Inventory Config’s report so we do not have to make an additional S3 call but we aggregate it the same way above.


Process:

Installation

There is one new parameter in the CFT: Enable Storage Assessment. If that is enabled the app will do a manual crawl on all of your buckets and put an enabled inventory config on all of your buckets. If that is disabled it will not do a manual crawl and will put an inventory config on all buckets in a disabled state. To enable Storage Assessment if you have disabled it in the CFT you will have to go to the Console Settings page and Enable the Storage Assessment toggle and then go to the Storage Assessment page and enable the buckets you would like to be aggregated. If you install it with Storage Assessment disabled a manual crawl will not happen, it will wait until the first Inventory Config report is generated and delivered.

Upgrade

On upgrade, Storage Assessment is enabled by default. This means it will first do a manual crawl on all of your buckets then put an enabled Inventory Config on your buckets.

Console Settings Page

To enable/disable Storage Assessment throughout the application toggle this setting on/off on the Console Settings Page. If you are enabling it you will then have to follow the link and enable the buckets you’d like aggregated.

Disclaimer

Enabling Storage Assessment on the Console Settings page will not enable any bucket’s Inventory Config, it just allows the application to continue the Storage Assessment process.

Storage Assessment Page

The settings modal in the top right of the page is where you can enable/disable buckets and configure your Percent Scanned settings. The first tab will display a table with buckets. Checked buckets are enabled (can also tell the buckets status by the Enabled column). To enable buckets check the box to the left of the bucket name and just click save. To disable buckets simply uncheck a checked box and click save. You can enable/disable multiple buckets at a time. Percent Scanned configuration contains an Enabled/Disabled toggle, a time frame selection, and a Run Tonight option. To configure the last two you need to enable Percent Scanned in the toggle. Time Frame is the amount of time between scans and if change will set the next date as X days from the current day. If Run Tonight is selected it will set the next Percent Scanned date as tonight and the time frame will be set accordingly.

storage assessment settings buckets

storage assessment settings percent scanned


Last update: January 10, 2023