Adam Jarret Blog

If an S3 bucket or CloudFront instance has access logging enabled, the logs will be delivered as plain text files to the designated target bucket.

Having access logs split into many files in an S3 bucket is not inherently useful but presumably Amazon takes this approach for the flexibility of processing options it affords. Downloading the logs to a local machine for analysis is a bit inelegant; since the logs are already “in the cloud” it is preferable to access them in a searchable manner via a web browser. CloudWatch is a product seemingly tailor made to solve this problem but unfortunately there is no turnkey solution to import access logs from S3. Luckily, it’s not too difficult to roll your own.

Enable Logging

The AWS documentation has instructions for enabling access logs for S3 and for CloudFront.

I recommend creating a single target bucket for all log files and specifying a different target prefix for each log source. For example, if you enable logging on two S3 buckets, specify the same target bucket for both but specify a unique target prefix for each. This allows a single Lambda function with a single trigger to handle all incoming logs.

Create IAM Policy

Use the IAM policies section of the AWS Console to create a new policy granting access to the target bucket and CloudWatch logs.

Click the Create policy button and select Create Your Own Policy. The policy document should look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::target-bucket-name/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
         }
    ]
}

Create IAM Role

Use the IAM roles section of the AWS Console to create a new role for this Lambda.

Click the Create role button and choose AWS Service > Lambda, then click Next.

Check the box next to the policy you created in the previous step and click Next.

Type a name for the role and click Create role.

Create Lambda Function

Use the Lambda section of the AWS Console to create a new Lambda function.

Click the Create function button and choose Author from scratch.

Type a name for the function, select the role you defined in the previous step and click Create function.

The Runtime should be Node.js 6.10 and the Handler should be index.handler (which are the defaults).

On the Triggers tab, click Add Trigger. Choose S3 from the list and select the target bucket. The Event type should be Object Created (All) and Prefix and Suffix should be blank.

The code for the function is available in this gist (which is embedded below):