If an S3 bucket or CloudFront instance has access logging enabled, the logs will be delivered as plain text files to the designated target bucket.
Having access logs split into many files in an S3 bucket is not inherently useful but presumably Amazon takes this approach for the flexibility of processing options it affords. Downloading the logs to a local machine for analysis is a bit inelegant; since the logs are already “in the cloud” it is preferable to access them in a searchable manner via a web browser. CloudWatch is a product seemingly tailor made to solve this problem but unfortunately there is no turnkey solution to import access logs from S3. Luckily, it’s not too difficult to roll your own.
The AWS documentation has instructions for enabling access logs for S3 and for CloudFront.
I recommend creating a single target bucket for all log files and specifying a different target prefix for each log source. For example, if you enable logging on two S3 buckets, specify the same target bucket for both but specify a unique target prefix for each. This allows a single Lambda function with a single trigger to handle all incoming logs.Pro Tip: The target prefix should end in a forward slash (/) so log files are grouped into “subfolders” by source. The example Lambda function below expects this and will use the first part of the log file path as the “log group” in CloudWatch.
Create IAM Policy
Use the IAM policies section of the AWS Console to create a new policy granting access to the target bucket and CloudWatch logs.
Click the Create policy button and select Create Your Own Policy. The policy document should look like this:
Create IAM Role
Use the IAM roles section of the AWS Console to create a new role for this Lambda.
Click the Create role button and choose AWS Service > Lambda, then click Next.
Check the box next to the policy you created in the previous step and click Next.
Type a name for the role and click Create role.
Create Lambda Function
Use the Lambda section of the AWS Console to create a new Lambda function.
Click the Create function button and choose Author from scratch.
Type a name for the function, select the role you defined in the previous step and click Create function.
The Runtime should be
Node.js 6.10 and the Handler should be
index.handler (which are the defaults).
On the Triggers tab, click Add Trigger. Choose S3 from the list and select the target bucket. The
Event type should be
Object Created (All) and Prefix and Suffix should be blank.
The code for the function is available in this gist (which is embedded below):