datahub-cloudformation
所属分类:云计算
开发工具:GO
文件大小:417053KB
下载次数:0
上传日期:2023-03-10 21:32:02
上 传 者:
sh-1993
说明: 使用AWS CloudFormation部署数据中心堆栈的存储库
(Repository for deploying datahub stack using AWS CloudFormation)
文件列表:
Ingestion (0, 2023-06-23)
Ingestion\acryl-executor-ecs-stack.sh (1438, 2023-06-23)
Ingestion\fargate-login.sh (816, 2023-06-23)
Ingestion\templates (0, 2023-06-23)
Ingestion\templates\python.ecs.template.yaml (25221, 2023-06-23)
license (0, 2023-06-23)
s3upload.sh (907, 2023-06-23)
scripts (0, 2023-06-23)
scripts\check_extra_tags-v2.sh (866, 2023-06-23)
scripts\check_extra_tags-v3.sh (939, 2023-06-23)
scripts\check_extra_tags.sh (857, 2023-06-23)
scripts\python-jwt.tar.gz (716, 2023-06-23)
scripts\python-jwt (0, 2023-06-23)
scripts\python-jwt\create_jwt.py (660, 2023-06-23)
scripts\python-jwt\requirements.txt (13, 2023-06-23)
scripts\remove_namespace_finalizer.sh (548, 2023-06-23)
templates (0, 2023-06-23)
templates\datahub-deployment-cleanup.yaml (26874, 2023-06-23)
templates\datahub-deployment-existing-vpc-2-subnets.yaml (35177, 2023-06-23)
templates\datahub-deployment-existing-vpc-3-subnets.yaml (32247, 2023-06-23)
templates\datahub-deployment-v2-existing-vpc-2-subnets.yaml (41884, 2023-06-23)
templates\datahub-deployment-v2-existing-vpc-3-subnets.yaml (42055, 2023-06-23)
templates\datahub-deployment-v2.yaml (46362, 2023-06-23)
templates\datahub-deployment.yaml (37412, 2023-06-23)
templates\nested (0, 2023-06-23)
templates\nested\2-subnets (0, 2023-06-23)
templates\nested\2-subnets\admin-v2.yaml (37541, 2023-06-23)
templates\nested\2-subnets\admin.yaml (33349, 2023-06-23)
templates\nested\2-subnets\eks-cluster.yaml (27174, 2023-06-23)
templates\nested\2-subnets\eks-nodegroup.yaml (7781, 2023-06-23)
templates\nested\2-subnets\elasticsearch-v2.yaml (5843, 2023-06-23)
templates\nested\2-subnets\elasticsearch.yaml (6060, 2023-06-23)
templates\nested\2-subnets\msk-v2.yaml (5123, 2023-06-23)
templates\nested\2-subnets\msk.yaml (5462, 2023-06-23)
templates\nested\2-subnets\mysql-v2.yaml (6083, 2023-06-23)
... ...
# Cloudformation Demo
![AcryDatahubCFN](https://user-images.githubusercontent.com/1105928/138394072-c86ddffa-5b6d-433f-95c8-37***842445d4.png)
## step 1 to 4 runs on customer AWS account
1. upload templates/scripts/license to S3
- get license file from Acryl, for example: <>
- clone this repo
- set AWS access, then upload needed files to your S3 bucket (for example,cf-templates-xxx-us-west-2), under folder 'dev'
```console
cd cloudformation
export AWS_PROFILE=***
./s3upload.sh cf-templates-xxx-us-west-2 dev xxx-customer1.yaml
```
2. create stack to deploy datahub platform in AWS
- choose region -> Cloudformation -> Create stack
- choose template based on your use case
- Template Amazon S3 URL: https://cf-templates-xxx-us-west-2.s3.us-west-2.amazonaws.com/dev/templates/datahub-deployment-v2.yaml (will create new VPC with 3 Subnets first, then deploy datahub to the new VPC)
- Template Amazon S3 URL: https://cf-templates-xxx-us-west-2.s3.us-west-2.amazonaws.com/dev/templates/datahub-deployment-v2-existing-vpc-3-subnets.yaml (will use existing VPC with 3 Subnets to deploy datahub)
- Template Amazon S3 URL: https://cf-templates-xxx-us-west-2.s3.us-west-2.amazonaws.com/dev/templates/datahub-deployment-v2-existing-vpc-2-subnets.yaml (will use existing VPC with 2 Subnets deploy datahub)
- StackName: dev-datahub
- TemplateBucketName: cf-templates-xxx-us-west-2
- Environment: dev
- VPCID: vpc-0xxxxxxxxxxxxxxxx
- The AZ's to deploy to: choose 3 or 2 AZs per your use case
- The Existing Private Subnet 1 ID: subnet-1xxxxxxxxxxxxxxxx
- The Existing Private Subnet 2 ID: subnet-2xxxxxxxxxxxxxxxx
- The Existing Private Subnet 3 ID: subnet-3xxxxxxxxxxxxxxxx, or leave empty for 2-subnet setup
- Enable Creation of ElasticSearch Service Role: set to true if ServiceLinked Role for ES doesn't exists
- The AWS IAM Role arn that will be allowed to manage EKS, for example: aws:iam::AccountID:role/admin-role
- DataHub Domain Name: datahub.a.b.c
- ELB cert arn: arn: arn:aws:acm:REGION:AccountID:certificate/xx-xx-xx-xx-xx (ssl cert for datahub.a.b.c)
- The Elastic Load Balancer Inbound CIDRs: comma seperated CIDR list that can access DataHub ALB
- Kots Admin Domain Name: kotsadm.e.f.g
- Kots ELB cert arn: arn:aws:acm:REGION:AccountID:certificate/xx-xx-xx-xx-xx (ssl cert for kotsadm.e.f.g)
- Applicaiton: Kots application name
- ApplicationReleaseChannel: choose Kots Application Release Channel
- CreatePrivateLink: Choose 'true' to create private link endpoint service
- RemoveTempResources: if you want to keep admin provision host, choose 'false'
- Tags, can have customized tags here, no space allowed in either Key or Value
- Stack failure options: choose 'rollback on failure'
- for troubleshooting, choose 'Preserve successfully provisioned resources' (useful when working on development of cloudformation)
- check:
- "I acknowledge that AWS CloudFormation might create IAM resources with custom names."
- "I acknowledge that AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND"
- click "Create stack"
- you will see a stack 'dev-datahub' (this is master stack), and it will invoke nested stacks in order
3. find datahub platform info
- after Stack Info show Status 'CREATE_COMPLETE', you can find needed info from nested stack <>'s Outputs
4. manually create DNS record
- create datahub.xxx.xxx.com to point to the datahub-frontend ALB
- create new routes to reach this new VPC
- access https://datahub.xxx.xxx.com to use datahub
## step 5 runs on Acryl AWS account
5. manually create VPC endpoint
- under Acryl AWS account, go to same region as customer, find service by service name, for example com.amazonaws.vpce.us-west-2.vpce-svc-*** (get service name from step 4.), select shared vpc, choose 3 private subnets, attach default security group
- access https://{vpc_endpoint_dns} to use manage datahub release
近期下载者:
相关文件:
收藏者: