Focusing on the customer is the right thing to do - every time. AWS Audit Manager was developed with AWS customers in mind - a service that AWS customers can use to automate evidence collection and prepare for an audit. This approach resulted in a pretty cool service, that I discussed at length in CISO Mag.
The challenge of focusing on customers is that it resulted in reporting functionality that doesn't benefit assessors or auditors. In traditional cybersecurity services or products, focusing on the customer is enough to solve the problem that your product or service was designed to solve. Unfortunately in the cybersecurity audit space, products and services must consider the customer and auditor in mind.
It is impossible to solve the problem of manual, slow, time-consuming audits without making things easier for the auditor too. An audit is a partnership between the company being audited and the auditor. If the auditor cannot use the evidence automatically-collected then they will revert to manual evidence requests to perform their assessments.
The customers (in this case AWS Customers) still need to hire an auditor or in SOC 2 examinations, a Certified Public Accountant (CPA) to perform the audit. These auditors have evidence requirements and testing requirements that need to be considered with a tool like this. In its current state, the AWS Audit Manager service does not help SOC 2 auditors perform their assessments, which will result in duplicate work for the company being audited.
While the output of the reporting is challenging for the auditor to use. It is equally manual, and time-consuming for the AWS user to generate the report.
Let's dive into how you generate a report and check out the outputs:
Generating a report requires a few setup steps. First things first, you need to decide which evidence will be included in the report. Ideally, all of the evidence collected, right? Unfortunately, there is no select all button here. The manual step can be time-consuming as it requires you to:
Navigate to each individual control
Navigate to the evidence folder, select the evidence you would like to include in the report:
Click add to the assessment report
Now, this may not seem like a lot of steps, but let's think about the SOC 2 framework within AWS Audit Manager. The SOC 2 framework includes 56 controls, 35 are manual and 21 are automated, according to AWS. Imagine completing this for 56 controls, the example above shows one piece of evidence for a criterion but think about the below scenario for CC6.1. There are 54 total pieces of evidence, over 83 resources evaluated. A bunch of different compliance check statuses which will require administrators to individually review each to ensure they should be included in the SOC 2 examination. This isn't quite "automated evidence" collection" as you probably imagined it:
Manually selecting each evidence item for each control area can become an entire day worth of tasks in and of itself.
Before we continue, there is also the issue here that compliance managers may or may not understand how these services relate to a particular criterion or understand the service being evaluated by AWS Audit Manager. For example:
So for CC3.1 above, AWS Audit Manager is telling the user that the only evidence needed from AWS for this criterion is that AWS Security Hub, Amazon Inspector, AWS Config, Amazon GuardDuty, and Amazon Macie are enabled in each region and account in-scope.
There are a few issues with this:
CC3.1 states "The entity specifies objectives with sufficient clarity to enable the identification and assessment of risks relating to objectives."
Another nice-to-have here would be an actual control statement and text of the criteria here, outlining what the criterion text states and what an example control statement would be to address the criteria.
It is not clear to the user how the testing information relates to CC3.1. In order for the user to know what CC3.1 includes, they must click on the PDF linked in the screenshot above and scroll through an AICPA document to find the text of the criterion.
The text of the criteria or control statement should be easy to add directly to the AWS Audit Manager console so compliance managers or security professionals don't need to memorize the SOC 2 criteria or have the AICPA documentation open to use the tool.
The other issue here is the testing information and lack of clarity from the service on how the testing information and evidence relate to the criterion. I dive deeper into the SOC 2 criteria mapping issues in this ByteChek Learning Center article.
The option to upload manual evidence is great because, on a criterion like CC3.1, manual evidence will be required here.
However, it is not clear how the relationship between the user (company) and their auditor will be facilitated on controls like this. In this instance, the evidence collected doesn't necessarily address CC3.1. Below is an excerpt from the SOC 2 Guide Points of Focus that the AICPA suggest you consider for CC3.1:
You can see there is a gap between what AWS Audit Manager is saying works for CC3.1 and what the AICPA says.
This would mean that either:
a) The client assumes enabling Config, Inspector, SecurityHub, GuardDuty, and Macie will address CC3.1. They then provide "evidence" from AWS Audit Manager to their auditor thinking this will suffice. Then the auditor will inform them they need more evidence to complete the audit ☹️.
b) The auditor has examined every criterion within AWS Audit Manager (not likely) and will inform the client where the automated evidence outlined within the AWS console is incorrect. This will then result in a separate evidence request list where the client (AWS user) will have to determine if they would like to upload the evidence to AWS Audit Manager or to their auditors' GRC tool.
Ok, back to the report. So once you've manually selected the "automated" or manual evidence associated with the criterion.
You generate the assessment report:
It only takes a few seconds for the report to generate and be ready for download under the Assessment Reports tab:
So what does the "audit-ready" report look like?
This "audit ready" report leaves a lot to be desired. So, let's dive into CC6.1 as another example.
As displayed above there are six pieces of evidence listed under CC6.1. Unfortunately, 5 out of those 6 pieces of evidence are not applicable to the assessment we are performing because the services being evaluated are not being used by the service organization.
Despite these pieces of evidence being "not applicable", the evidence is downloaded into the audit report zip file. Once the auditor dives into the evidence folder it is not clear what's going on. Here is an example piece of evidence:
This is the output of the evidence from AWS Audit Manager. I understand the intent and a lot of the items on this evidence PDF make sense to me. However, if your auditor is not familiar with AWS terminology and syntax, this could be a very confusing document.
The reason this is important is that the tool will only be useful if the outputs can be used to help organizations complete their audits. The benefits of automated evidence collection will not be realized if organizations still have to manually produce evidence for their auditors because the output is not useful or readable by the auditors.
I have a lot of optimism for AWS Audit Manager but the biggest weakness and gap in the tool is the reporting functionality. The promise of an "audit-ready" report was not fulfilled which can result in confusion and significant back and forth between the auditor and client being audited.
At ByteChek, we understand the inputs and outputs needed for a SOC 2 examination. In fact, our co-founder helped author the SOC 2 Guide. Our platform includes functionality to generate key aspects of your SOC 2 report such as the dreaded system description. Chat with us using the bot below to learn more.