Datasets
Models
Downloads: 1
Switch to unified view
| a | b/deployment/genomics-tertiary-analysis-and-data-lakes-using-aws-glue-and-amazon-athena.template | ||
|---|---|---|---|
| 1 | AWSTemplateFormatVersion: '2010-09-09' |
||
| 2 | |||
| 3 | Description: | |
||
| 4 | (SO0077) - This Guidance for Multi-Omics and Multi-Modal Data Integration and Analysis on AWS creates a scalable environment in AWS to prepare genomic, clinical, mutation, expression and imaging data for large-scale analysis and perform interactive queries against a data lake. This solution demonstrates how to 1)Provision Amazon Omics resources to ingest, store and query genomics data, 2) provision serverless data ingestion pipelines for multi-modal data preparation and cataloging, 3) visualize and explore clinical data through an interactive interface, and 4) run interactive analytic queries against a multi-modal data lake. Version v3.0.0. |
||
| 5 | |||
| 6 | Mappings: |
||
| 7 | Send: |
||
| 8 | AnonymousUsage: |
||
| 9 | Data: Yes |
||
| 10 | SourceCode: |
||
| 11 | General: |
||
| 12 | S3Bucket: '%%BUCKET_NAME%%' |
||
| 13 | KeyPrefix: '%%SOLUTION_NAME%%/%%VERSION%%' |
||
| 14 | |||
| 15 | Parameters: |
||
| 16 | Project: |
||
| 17 | Type: String |
||
| 18 | Description: > |
||
| 19 | The project name for this solution. The project name will be used to prefix resources created by this solution. The solution Glue database name and Athena workgroup name will be the project name. Project names should be unique to a project. |
||
| 20 | AllowedPattern: "[a-zA-Z0-9-]{3,24}" |
||
| 21 | ConstraintDescription: > |
||
| 22 | Project name should be unique, 3-24 characters in length, and only have alphanumeric characters and hyphens ([a-zA-Z0-9-]{3,32}). |
||
| 23 | Default: GenomicsAnalysis |
||
| 24 | |||
| 25 | Resources: |
||
| 26 | Setup: |
||
| 27 | Type: Custom::Setup |
||
| 28 | DependsOn: |
||
| 29 | - CodeBuild |
||
| 30 | Version: 1.0 |
||
| 31 | Properties: |
||
| 32 | ServiceToken: !Sub ${SetupLambda.Arn} |
||
| 33 | CodeBuildProjectName: !Sub ${CodeBuild} |
||
| 34 | |||
| 35 | SetupLambda: |
||
| 36 | Type: AWS::Lambda::Function |
||
| 37 | DependsOn: |
||
| 38 | - SetupLambdaRole |
||
| 39 | Properties: |
||
| 40 | Handler: lambda.handler |
||
| 41 | Runtime: python3.8 |
||
| 42 | FunctionName: !Sub ${Project}Setup |
||
| 43 | Code: |
||
| 44 | S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] |
||
| 45 | S3Key: !Join ["", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "/SolutionSetup.zip"]] |
||
| 46 | Role: !Sub ${SetupLambdaRole.Arn} |
||
| 47 | Timeout: 600 |
||
| 48 | Metadata: |
||
| 49 | cfn_nag: |
||
| 50 | rules_to_suppress: |
||
| 51 | - id: W58 |
||
| 52 | reason: Bug in CfnNag. |
||
| 53 | - id: W89 |
||
| 54 | reason: Lambda only used on setup. |
||
| 55 | - id: W92 |
||
| 56 | reason: No need for concurrent execution. |
||
| 57 | SetupLambdaRole: |
||
| 58 | Type: AWS::IAM::Role |
||
| 59 | DependsOn: |
||
| 60 | - CodeBuild |
||
| 61 | Properties: |
||
| 62 | AssumeRolePolicyDocument: |
||
| 63 | Version: 2012-10-17 |
||
| 64 | Statement: |
||
| 65 | - Action: |
||
| 66 | - sts:AssumeRole |
||
| 67 | Effect: Allow |
||
| 68 | Principal: |
||
| 69 | Service: |
||
| 70 | - lambda.amazonaws.com |
||
| 71 | Path: / |
||
| 72 | Policies: |
||
| 73 | - PolicyName: LogsAccess |
||
| 74 | PolicyDocument: |
||
| 75 | Statement: |
||
| 76 | - Effect: Allow |
||
| 77 | Action: |
||
| 78 | - logs:CreateLogGroup |
||
| 79 | - logs:CreateLogStream |
||
| 80 | - logs:PutLogEvents |
||
| 81 | Resource: |
||
| 82 | - !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${Project}* |
||
| 83 | - PolicyName: CodeBuildAccess |
||
| 84 | PolicyDocument: |
||
| 85 | Statement: |
||
| 86 | - Effect: Allow |
||
| 87 | Action: |
||
| 88 | - codebuild:BatchGetProjects |
||
| 89 | - codebuild:BatchGetBuilds |
||
| 90 | - codebuild:StartBuild |
||
| 91 | Resource: |
||
| 92 | - !Sub ${CodeBuild.Arn} |
||
| 93 | - PolicyName: EventsAccess |
||
| 94 | PolicyDocument: |
||
| 95 | Statement: |
||
| 96 | - Effect: Allow |
||
| 97 | Action: |
||
| 98 | - events:DeleteRule |
||
| 99 | - events:PutRule |
||
| 100 | - events:PutTargets |
||
| 101 | - events:RemoveTargets |
||
| 102 | Resource: |
||
| 103 | - !Sub arn:aws:events:${AWS::Region}:${AWS::AccountId}:rule/Setup* |
||
| 104 | - PolicyName: LambdaAccess |
||
| 105 | PolicyDocument: |
||
| 106 | Statement: |
||
| 107 | - Effect: Allow |
||
| 108 | Action: |
||
| 109 | - lambda:AddPermission |
||
| 110 | - lambda:RemovePermission |
||
| 111 | Resource: |
||
| 112 | - !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${Project}* |
||
| 113 | |||
| 114 | CodeBuildRole: |
||
| 115 | Type: AWS::IAM::Role |
||
| 116 | Properties: |
||
| 117 | AssumeRolePolicyDocument: |
||
| 118 | Version: 2012-10-17 |
||
| 119 | Statement: |
||
| 120 | - Action: |
||
| 121 | - sts:AssumeRole |
||
| 122 | Effect: Allow |
||
| 123 | Principal: |
||
| 124 | Service: |
||
| 125 | - codebuild.amazonaws.com |
||
| 126 | Path: / |
||
| 127 | Policies: |
||
| 128 | - PolicyName: CloudFormationAccess |
||
| 129 | PolicyDocument: |
||
| 130 | Statement: |
||
| 131 | - Action: |
||
| 132 | - cloudformation:CreateStack |
||
| 133 | - cloudformation:DescribeStacks |
||
| 134 | - cloudformation:DescribeStackResource |
||
| 135 | - cloudformation:DescribeStackResources |
||
| 136 | - cloudformation:UpdateStack |
||
| 137 | - cloudformation:DeleteStack |
||
| 138 | - cloudformation:UpdateTerminationProtection |
||
| 139 | Effect: Allow |
||
| 140 | Resource: !Sub arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/${Project}* |
||
| 141 | - PolicyName: LogsAccess |
||
| 142 | PolicyDocument: |
||
| 143 | Statement: |
||
| 144 | - Effect: Allow |
||
| 145 | Action: |
||
| 146 | - logs:CreateLogGroup |
||
| 147 | - logs:CreateLogStream |
||
| 148 | - logs:PutLogEvents |
||
| 149 | Resource: |
||
| 150 | - !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/${Project}* |
||
| 151 | - PolicyName: IAMAccess |
||
| 152 | PolicyDocument: |
||
| 153 | Statement: |
||
| 154 | - Effect: Allow |
||
| 155 | Action: |
||
| 156 | - iam:CreateRole |
||
| 157 | - iam:DeleteRole |
||
| 158 | - iam:PutRolePolicy |
||
| 159 | - iam:DeleteRolePolicy |
||
| 160 | - iam:AttachRolePolicy |
||
| 161 | - iam:DetachRolePolicy |
||
| 162 | - iam:UpdateAssumeRolePolicy |
||
| 163 | - iam:PassRole |
||
| 164 | - iam:GetRole |
||
| 165 | - iam:GetInstanceProfile |
||
| 166 | - iam:CreateInstanceProfile |
||
| 167 | - iam:DeleteInstanceProfile |
||
| 168 | - iam:AddRoleToInstanceProfile |
||
| 169 | - iam:RemoveRoleFromInstanceProfile |
||
| 170 | Resource: |
||
| 171 | - !Sub arn:aws:iam::${AWS::AccountId}:role/${Project}* |
||
| 172 | - !Sub arn:aws:iam::${AWS::AccountId}:instance-profile/${Project}* |
||
| 173 | - !Sub arn:aws:iam::${AWS::AccountId}:role/aws-quicksight-service-role-v0 |
||
| 174 | - PolicyName: CodeBuildAccess |
||
| 175 | PolicyDocument: |
||
| 176 | Statement: |
||
| 177 | - Effect: Allow |
||
| 178 | Action: |
||
| 179 | - codebuild:CreateProject |
||
| 180 | - codebuild:UpdateProject |
||
| 181 | - codebuild:ListProjects |
||
| 182 | - codebuild:BatchGetProjects |
||
| 183 | - codebuild:DeleteProject |
||
| 184 | Resource: |
||
| 185 | - !Sub arn:aws:codebuild:${AWS::Region}:${AWS::AccountId}:project/${Project}* |
||
| 186 | - PolicyName: CodePipelineAccess |
||
| 187 | PolicyDocument: |
||
| 188 | Statement: |
||
| 189 | - Effect: Allow |
||
| 190 | Action: |
||
| 191 | - codepipeline:CreatePipeline |
||
| 192 | - codepipeline:GetPipeline |
||
| 193 | - codepipeline:UpdatePipeline |
||
| 194 | - codepipeline:DeletePipeline |
||
| 195 | - codepipeline:GetPipelineState |
||
| 196 | - codepipeline:ListPipelineExecutions |
||
| 197 | Resource: |
||
| 198 | - !Sub arn:aws:codepipeline:${AWS::Region}:${AWS::AccountId}:${Project}* |
||
| 199 | - PolicyName: CodeCommitAccess |
||
| 200 | PolicyDocument: |
||
| 201 | Statement: |
||
| 202 | - Effect: Allow |
||
| 203 | Action: |
||
| 204 | - codecommit:CreateBranch |
||
| 205 | - codecommit:CreateRepository |
||
| 206 | - codecommit:GetRepository |
||
| 207 | - codecommit:DeleteRepository |
||
| 208 | - codecommit:CreateCommit |
||
| 209 | - codecommit:GitPush |
||
| 210 | - codecommit:GitPull |
||
| 211 | - codecommit:DeleteBranch |
||
| 212 | Resource: |
||
| 213 | - !Sub arn:aws:codecommit:${AWS::Region}:${AWS::AccountId}:${Project}* |
||
| 214 | - Effect: Allow |
||
| 215 | Action: |
||
| 216 | - codecommit:ListRepositories |
||
| 217 | Resource: '*' |
||
| 218 | - PolicyName: EventsAccess |
||
| 219 | PolicyDocument: |
||
| 220 | Statement: |
||
| 221 | - Effect: Allow |
||
| 222 | Action: |
||
| 223 | - events:DescribeRule |
||
| 224 | - events:PutRule |
||
| 225 | - events:DeleteRule |
||
| 226 | - events:PutTargets |
||
| 227 | - events:RemoveTargets |
||
| 228 | Resource: |
||
| 229 | - !Sub arn:aws:events:${AWS::Region}:${AWS::AccountId}:rule/* |
||
| 230 | - PolicyName: GlueAccess |
||
| 231 | PolicyDocument: |
||
| 232 | Statement: |
||
| 233 | - Effect: Allow |
||
| 234 | Action: |
||
| 235 | - glue:StartCrawler |
||
| 236 | - glue:GetCrawlers |
||
| 237 | - glue:StartWorkflowRun |
||
| 238 | Resource: '*' |
||
| 239 | - PolicyName: LambdaAccess |
||
| 240 | PolicyDocument: |
||
| 241 | Statement: |
||
| 242 | - Effect: Allow |
||
| 243 | Action: |
||
| 244 | - lambda:GetFunction |
||
| 245 | - lambda:CreateFunction |
||
| 246 | - lambda:DeleteFunction |
||
| 247 | - lambda:InvokeFunction |
||
| 248 | Resource: |
||
| 249 | - !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${Project}* |
||
| 250 | - PolicyName: S3Access |
||
| 251 | PolicyDocument: |
||
| 252 | Statement: |
||
| 253 | - Effect: Allow |
||
| 254 | Action: |
||
| 255 | - s3:GetObject |
||
| 256 | Resource: |
||
| 257 | !Join |
||
| 258 | - '' |
||
| 259 | - - 'arn:aws:s3:::' |
||
| 260 | - !Join |
||
| 261 | - '-' |
||
| 262 | - - !FindInMap ["SourceCode", "General", "S3Bucket"] |
||
| 263 | - Ref: "AWS::Region" |
||
| 264 | - '/*' |
||
| 265 | - Effect: Allow |
||
| 266 | Action: |
||
| 267 | - s3:ListBucket |
||
| 268 | Resource: |
||
| 269 | !Join |
||
| 270 | - '' |
||
| 271 | - - 'arn:aws:s3:::' |
||
| 272 | - !Join |
||
| 273 | - '-' |
||
| 274 | - - !FindInMap ["SourceCode", "General", "S3Bucket"] |
||
| 275 | - Ref: "AWS::Region" |
||
| 276 | |||
| 277 | - Effect: Allow |
||
| 278 | Action: |
||
| 279 | - s3:PutObjectAcl |
||
| 280 | - s3:GetObject |
||
| 281 | - s3:PutObject |
||
| 282 | - s3:DeleteObject |
||
| 283 | - s3:ListBucket |
||
| 284 | - s3:CreateBucket |
||
| 285 | - s3:DeleteBucket |
||
| 286 | - s3:PutEncryptionConfiguration |
||
| 287 | - s3:PutBucketPublicAccessBlock |
||
| 288 | - s3:PutBucketLogging |
||
| 289 | - s3:PutBucketAcl |
||
| 290 | - s3:PutBucketOwnershipControls |
||
| 291 | Resource: |
||
| 292 | - arn:aws:s3:::*pipe* |
||
| 293 | - arn:aws:s3:::*pipe*/* |
||
| 294 | - Effect: Allow |
||
| 295 | Action: |
||
| 296 | - s3:CreateBucket |
||
| 297 | - s3:DeleteBucket |
||
| 298 | - s3:ListBucket |
||
| 299 | - s3:PutEncryptionConfiguration |
||
| 300 | - s3:PutBucketPublicAccessBlock |
||
| 301 | - s3:PutBucketLogging |
||
| 302 | - s3:PutBucketAcl |
||
| 303 | - s3:PutObject |
||
| 304 | - s3:PutObjectAcl |
||
| 305 | - s3:PutBucketOwnershipControls |
||
| 306 | Resource: |
||
| 307 | - arn:aws:s3:::*pipe* |
||
| 308 | - arn:aws:s3:::*pipe*/* |
||
| 309 | |||
| 310 | Metadata: |
||
| 311 | cfn_nag: |
||
| 312 | rules_to_suppress: |
||
| 313 | - id: W11 |
||
| 314 | reason: Star required for codecommit:ListRepositories and Glue actions. |
||
| 315 | |||
| 316 | CodeBuild: |
||
| 317 | Type: AWS::CodeBuild::Project |
||
| 318 | Properties: |
||
| 319 | Name: !Sub ${Project}Setup |
||
| 320 | Artifacts: |
||
| 321 | Type: NO_ARTIFACTS |
||
| 322 | Source: |
||
| 323 | Type: NO_SOURCE |
||
| 324 | BuildSpec: !Sub | |
||
| 325 | version: 0.2 |
||
| 326 | phases: |
||
| 327 | install: |
||
| 328 | commands: |
||
| 329 | - git config --global user.name automated_user |
||
| 330 | - git config --global user.email automated_email |
||
| 331 | - git config --global credential.helper '!aws codecommit credential-helper $@' |
||
| 332 | - git config --global credential.UseHttpPath true |
||
| 333 | - aws s3 cp s3://$ARTIFACT_BUCKET/$ARTIFACT_KEY_PREFIX/Solution.zip . |
||
| 334 | - unzip Solution.zip |
||
| 335 | - ./$SOLUTION_ACTION.sh |
||
| 336 | Environment: |
||
| 337 | ComputeType: BUILD_GENERAL1_SMALL |
||
| 338 | EnvironmentVariables: |
||
| 339 | - Name: SOLUTION_ACTION |
||
| 340 | Value: setup |
||
| 341 | - Name: PROJECT_NAME |
||
| 342 | Value: !Ref Project |
||
| 343 | - Name: ARTIFACT_BUCKET |
||
| 344 | Value: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] |
||
| 345 | - Name: ARTIFACT_KEY_PREFIX |
||
| 346 | Value: !FindInMap ["SourceCode", "General", "KeyPrefix"] |
||
| 347 | Image: aws/codebuild/standard:6.0 |
||
| 348 | Type: LINUX_CONTAINER |
||
| 349 | ServiceRole: !Sub ${CodeBuildRole} |
||
| 350 | TimeoutInMinutes: 60 |
||
| 351 | Metadata: |
||
| 352 | cfn_nag: |
||
| 353 | rules_to_suppress: |
||
| 354 | - id: W32 |
||
| 355 | reason: Customer can enable encryption if desired. |