Storage Assessment
The Storage Assessment page gives you detailed information about your Amazon S3 storage state.
Overview
Details that are useful for understanding your Amazon S3 environment and to give you an overall feel of your data include:
Relevant charts showing an S3 Overview
File Information
Bucket Information
Trends
You can filter the information shown by bucket, region, account, and/or date. You can also see the date of the last scan under the Storage Assessment Data section. Finally, under the Filters you can see the time frame you are viewing data for.
Top Row Informationals
The top row of widgets gives you a quick view to specific details around your S3 environment. The top row gives you the aggregate totals of the latest snapshot for Total Data in your S3 environment, Total Objects in your S3 environment, Buckets and Regions enabled with Storage Assessment, Total Objects Scanned with our application, and Total Objects Encrypted in your S3 environment.
Assessment Visuals
Any chart labeled Top X will show 10 if there are more than 10 total buckets in your S3 account. Additionally, the first 2 panel charts will only show you information about the latest snapshot. For example, if you have a 30 day time frame selected it will only display information about the latest day (yesterday) as that contains the most up to date information.
S3 Overview
This panel contains 4 charts about your S3 environment. Going clockwise from the top left: Total Object Count and Size chart of your current S3 environment, Top 10 Largest Buckets in GB's, Percent Scanned of 10 Largest Buckets, and Percent Encrypted of 10 Largest Buckets.
File Information
The panel contains 3 charts about files in your S3 environment. Going clockwise from the top left: Top 10 File Types by size, Top 10 File Types by count, and File Ages of all files in your S3 environment.
Bucket Information¶
This panel contains 4 charts about Buckets in your S3 environment. Going clockwise from the top left: Public and Private Bucket Count by Region Encrypted and Unencrypted Bucket Count by Region, Percentage of a regions scanned object, and Percentage of a regions encrypted objects. The top 2 charts information we get from our Bucket information collection and the bottom 2 charts information we get from Storage Assessment data. If you do not have Percent Scanned enabled that graph will display 0's for all regions.
Trends
The trends tab contains 3 graphs that will only be visible if your time frame is larger than 1 day. They will display trends for whatever time frame is selected. First Graph is Bucket Object count by day, then Bucket Size by day, then Percent Scanned of each bucket by day.
Keywords
Storage Assessment
An assessment of aggregated stats by our application using a manual bucket crawl initially and then AWS’s Inventory configuration report for the remaining time.
Manual Crawl
Our application manually crawls your buckets and gathers information about the files sizes, last modified date, encryption status, and file type.
Inventory Configuration
A snap shot report generated once a day by AWS that contains information about your bucket contents. It contains file name, size, last modified date, encryption status, storage class, intelligent tiering access tier, and the name the bucket it resides in. Inventory configuration can have 2 status: Enabled and Disabled. Enabled means that a report is generated and delivered every night. Disabled means that Inventory Configuration settings are primed on the bucket but no report is generated and delivered. Our application configures the Inventory Config to deliver the reports to a directory in your quarantine bucket in the same buckets region.
Percent Scanned
This refers to the percent of objects scanned in your bucket. It takes the key name from the Inventory Config and checks the file for tags in S3 that match our applications tags. We then take the count of objects with matching tags and find the percentage against your buckets total object count.
Since the Inventory Config’s report is generated separately from when we aggregate it there could be a discrepancy if files are deleted during that time frame that are in the report.
Percent Encrypted
We can get the file encryption status information from the Inventory Config’s report so we do not have to make an additional S3 call but we aggregate it the same way above.
Requirements for Storage Assessment
For Storage Assessment to be active it must be:
Enabled on the Console Settings page (under Configuration).
At the Storage Assessment Settings modal's Enable Bucket Assessments tab each bucket that you want to include in the report must be enabled.
If you want to have an Objects Scanned count and percentage, then you must enable Percent Scanned Settings at the Storage Assessment Settings modal. More details can be found in the Process section of this help page.
Enabling or Disabling
There is one new parameter in the CloudFormation Template used for deployment: Enable Storage Assessment.
If that parameter is enabled the app will do a manual crawl on all of your buckets and put an enabled inventory config on all of your buckets. If that parameter is disabled it will not do a manual crawl and will put an inventory config on all buckets in a disabled state.
To enable Storage Assessment if you have disabled it in the CloudFormation Template you will have to go to the Console Settings page, enable the Storage Assessment toggle, and then go to the Storage Assessment page to enable the buckets you would like to be aggregated. If you install the app with Storage Assessment disabled a manual crawl will not happen, it will wait until the first Inventory Config report is generated and delivered.
Console Settings Page
To enable/disable Storage Assessment throughout the application toggle this setting on/off on the Console Settings Page. If you are enabling it you will then have to follow the link and enable the buckets you’d like aggregated.
Enabling Storage Assessment on the Console Settings page will not enable any bucket’s Inventory Config, it just allows the application to continue the Storage Assessment process.
If you're upgrading your deployment
Upon upgrade, Storage Assessment is enabled by default. This means it will first do a manual crawl on all of your buckets then put an enabled Inventory Config on your buckets.
Assessment Settings
The settings modal in the top right of the page is where you can enable/disable buckets and configure your Percent Scanned settings.
Enable Bucket Assignments is the first tab and it will display a table with buckets. Checked buckets are enabled (can also tell the buckets status by the Enabled column). To enable buckets check the box to the left of the bucket name and just click save. To disable buckets simply uncheck a checked box and click save. You can enable/disable multiple buckets at a time. time frame will be set accordingly.
On the Bucket Protection page Event Driven Scanning is off for all buckets at deployment (because its relatively expensive), and on this page Bucket Assessments are enabled for all buckets at deployment (because it's relatively cheap), but you may want to put these settings in sync with each other.
Percent Scanned Settings is the second tab and this configuration contains
An Enabled/Disabled toggle
A time frame selection
A Run Tonight option
To configure the last two you need to enable Percent Scanned in the toggle.
Time Frame is the amount of time between scans and if change will set the next date as X days from the current day. If Run Tonight is selected it will set the next Percent Scanned date as tonight and the time frame will be set accordingly.
Additional Costs
This feature is automatically turned on when you upgrade to the latest release through the standard console upgrade process. You can disable this feature at upgrade by manually updating the stack and turning the drop down selection to False.
This feature will start by crawling all of your data to give you an initial overview to all of the files you have stored in Amazon S3. After that initial crawl, Storage Assessment will leverage S3 Inventory to continue to assess the data. Both the crawl and S3 Inventory have minimal charges associated with them (but they could add up over a month if you do this daily).
Percent Scanned requires the checking of S3 Tags to determine if the scan-result tags exist on the objects. GetObjectTagging calls will be made against every inventoried S3 bucket. Please take into account the following pricing considerations for Storage Assessment:
The initial crawl will invoke LIST calls to gather all of the objects associated with your S3 buckets. AWS charges $0.005 per thousand list calls and you receive 1000 objects per list call. This means crawling 1 million objects costs $0.005.
Storage Assessment generates nightly (in the future this will be configurable) S3 inventory reports. AWS charges $0.0025 per million objects listed for an S3 inventory report.
If you want to calculate the number/percent of objects scanned this functionality performs GET calls for each object. GET calls cost $0.40 per million objects. The frequency of this calculation is configurable within the Storage Assessment settings.
Last updated
