shentufoundation/shentu

[operation]Make the S3 buckets shentu-data-snapshot-full and shentu-data-snapshot-light public accessible

haozhan9 opened this issue · 1 comments

Since the fixing of AWS findings, these two buckets are not simply public available any more. This issue is created to trace the work of making them public accessible through the cloudfront. The recommendation from platform guy is pasted below:

You shouldn't be connecting to the s3 endpoint directly even for public content. For public content you should set up:
Cloudfront -> [Resource policy to restrict Cloudfront via OAC] S3 bucket
There are a lot of advantages to fronting your public content via Cloudfront instead of going directly to the bucket, I'll just list off a few:
This proves that you made a conscious decision to make your content public. It is hard for the security team to keep up with what buckets should be public and what should not be (just in Certik we have 36 AWS accounts) so AWS recommends blocking all public access to s3 and enforcing everyone to do this extra step proving they want their content to be public
Cloudfront is a CDN, this will cache your objects on edge nodes making end user experience better by access objects faster
In the future, security team might use web application firewall (WAF) to do things like prevent access from known abuse IPs (ie comprised machines). It is best practice for all content exposed to the internet to be filtered by WAF. We cannot put a WAF in front of the S3 public endpoint but can do it in front of Cloudfront
Informational: Client -> Cloudfront -> S3 connection path is free so there is no cost concern
S3 does not allow custom domain but cloudfront allows you to add a custom domain, which you should be using. In a Organization, we should never use domains in production that we don't own (ie .s3.region.amazonaws.com or .cloudfront.net ). Say you want to go multi-region, move how your data is served, or need to move your data to a different AWS account; you don't want to be tied down the a domain you cannot move because then you need to tell everyone, since this is public content, to change the URL they are using. You should use a custom domain, which by the way every account has a domain it can use, to just repoint to the new location so your end user is not affected.

Here is the operation process given by AWS.

Url for snapshot-light
https://d35kzm6d1vecp9.cloudfront.net/shentu.tar.gz