aws-parallelcluster:AWS ParallelCluster是AWS支持的开源集群管理工具,用于在AWS云中部

  • C4_748306
  • 2.3MB
  • zip
  • 0
  • VIP专享
  • 0
  • 2022-06-15 08:43
AWS ParallelCluster-适用于云的HPC AWS ParallelCluster是AWS支持的开源集群管理工具,可让您轻松地在AWS云中部署和管理高性能计算(HPC)集群。 AWS ParallelCluster建立在开源CfnCluster项目的基础上,使您能够在AWS中快速构建HPC计算环境。 它会自动设置所需的计算资源和共享文件系统,并提供各种批处理调度程序,例如AWS Batch,SGE,Torque和Slurm。 AWS ParallelCluster促进了概念的快速启动证明(POC)和生产部署。 您可以在AWS ParallelCluster之上构建更高级别的工作流程,例如使整个DNA测序工作流程自动化的Genomics门户。 快速开始 重要信息:您将需要一个Amazon EC2密钥对,才能完成以下步骤。 请参阅 。 首先,请确保您已安装 : $ pip
# AWS ParallelCluster Integration Testing Framework The framework used to implement and run integration tests for AWS ParallelCluster is made of two main components: * **Integration Tests Orchestrator**: is a cli that allows to submit the integration tests. It takes care of setting up the test environment, it orchestrates parallel tests execution and generates the final tests reports. * **Integration Tests Framework**: the actual testing framework, based on pytest, which defines a series of fixtures and markers that allow to parametrize tests execution across several dimensions, to easily manage clusters lifecycle and re-usage and to perform cleanup on failures. It also offers a set of utility functions that implement features which are common to all tests such as remote command execution and ParallelCluster config generation. ![integtests_small]( ## Run Integration Tests ### Requirements To run the integration tests you have to use Python >= 3.7. Before executing integration tests it is required to install all the python dependencies required by the framework. In order to do that simply run the following command: ```bash pip install -r tests/integration-tests/requirements.txt ``` After that you can run the CLI by simply executing the following ``` cd tests/integration-tests python -m test_runner --help ``` ### The test_runner CLI The test_runner CLI is the main entry point to be used in order to run tests. Here is the output of the helper function that lists all the available options: ``` python -m test_runner --help usage: [-h] --key-name KEY_NAME --key-path KEY_PATH [-n PARALLELISM] [--sequential] [--credential CREDENTIAL] [--retry-on-failures] [--tests-root-dir TESTS_ROOT_DIR] [-c TESTS_CONFIG] [-i [INSTANCES [INSTANCES ...]]] [-o [OSS [OSS ...]]] [-s [SCHEDULERS [SCHEDULERS ...]]] [-r [REGIONS [REGIONS ...]]] [-f FEATURES [FEATURES ...]] [--show-output] [--reports {html,junitxml,json,cw} [{html,junitxml,json,cw} ...]] [--cw-region CW_REGION] [--cw-namespace CW_NAMESPACE] [--cw-timestamp-day-start] [--output-dir OUTPUT_DIR] [--custom-node-url CUSTOM_NODE_URL] [--custom-cookbook-url CUSTOM_COOKBOOK_URL] [--createami-custom-cookbook-url CREATEAMI_CUSTOM_COOKBOOK_URL] [--createami-custom-node-url CREATEAMI_CUSTOM_NODE_URL] [--custom-template-url CUSTOM_TEMPLATE_URL] [--custom-hit-template-url CUSTOM_HIT_TEMPLATE_URL] [--custom-awsbatchcli-url CUSTOM_AWSBATCHCLI_URL] [--custom-ami CUSTOM_AMI] [--pre-install PRE_INSTALL] [--post-install POST_INSTALL] [--benchmarks] [--benchmarks-target-capacity BENCHMARKS_TARGET_CAPACITY] [--benchmarks-max-time BENCHMARKS_MAX_TIME] [--vpc-stack VPC_STACK] [--cluster CLUSTER] [--no-delete] [--keep-logs-on-cluster-failure] [--keep-logs-on-test-failure] [--stackname-suffix STACKNAME_SUFFIX] [--dry-run] Run integration tests suite. optional arguments: -h, --help show this help message and exit --key-name KEY_NAME Key to use for EC2 instances (default: None) --key-path KEY_PATH Path to the key to use for SSH connections (default: None) -n PARALLELISM, --parallelism PARALLELISM Tests parallelism for every region. (default: None) --sequential Run tests in a single process. When not specified tests will spawn a process for each region under test. (default: False) --credential CREDENTIAL STS credential to assume when running tests in a specific region.Credentials need to be in the format <region>,<endpoint>,<aRN rel='nofollow' onclick='return false;'>,<externalId> and can be specified multiple times. <region> represents the region credentials are used for, <endpoint> is the sts endpoint to contact in order to assume credentials, <account-id rel='nofollow' onclick='return false;'> is the id of the account where the role to assume is defined, <externalId> is the id to use when assuming the role. (e.g. ap-east-1,,arn:aws:iam::<account-id rel='nofollow' onclick='return false;'>:role/role-to-assume,externalId) (default: None) --retry-on-failures Retry once more the failed tests after a delay of 60 seconds. (default: False) --tests-root-dir TESTS_ROOT_DIR Root dir where integration tests are defined (default: ./tests) Test dimensions: -c TESTS_CONFIG, --tests-config TESTS_CONFIG Config file that specifies the tests to run and the dimensions to enable for each test. Note that when a config file is used the following flags are ignored: instances, regions, oss, schedulers. Refer to the docs for further details on the config format. (default: None) -i [INSTANCES [INSTANCES ...]], --instances [INSTANCES [INSTANCES ...]] AWS instances under test. Ignored when tests-config is used. (default: []) -o [OSS [OSS ...]], --oss [OSS [OSS ...]] OSs under test. Ignored when tests-config is used. (default: []) -s [SCHEDULERS [SCHEDULERS ...]], --schedulers [SCHEDULERS [SCHEDULERS ...]] Schedulers under test. Ignored when tests-config is used. (default: []) -r [REGIONS [REGIONS ...]], --regions [REGIONS [REGIONS ...]] AWS regions where tests are executed. Ignored when tests-config is used. (default: []) -f FEATURES [FEATURES ...], --features FEATURES [FEATURES ...] Run only tests for the listed features. Prepending the not keyword to the feature name causes the feature to be excluded. (default: ) Test reports: --show-output Do not redirect tests stdout to file. Not recommended when running in multiple regions. (default: None) --reports {html,junitxml,json,cw} [{html,junitxml,json,cw} ...] create tests report files. junitxml creates a junit-xml style report file. html creates an html style report file. json creates a summary with details for each dimensions. cw publishes tests metrics into CloudWatch (default: []) --cw-region CW_REGION Region where to publish CloudWatch metrics (default: us-east-1) --cw-namespace CW_NAMESPACE CloudWatch namespace where to publish metrics (default: ParallelCluster/IntegrationTests) --cw-timestamp-day-start CloudWatch metrics pushed with at timestamp equal to the start of the current day (midnight) (default: False) --output-dir OUTPUT_DIR Directory where tests outputs are generated (default: tests_outputs) Custom packages and templates: --custom-node-url CUSTOM_NODE_URL URL to a custom node package. (default: None) --custom-cookbook-url CUSTOM_COOKBOOK_URL URL to a custom cookbook package. (default: None) --createami-custom-cookbook-url CREATEAMI_CUSTOM_COOKBOOK_URL URL to a custom cookbook package for the createami command. (default: None) --createami-custom-node-url CREATEAMI_CUSTOM_NODE_URL URL to a custom node package for the createami command. (default: None) --custom-template-url CUSTOM_TEMPLATE_URL URL to a custom cfn template. (default: None) --custom-hit-template-url CUSTOM_HIT_TEMPLATE_URL URL to a custom hit cfn template. (default: None) --custom-awsbatchcli-url CUSTOM_AWSBATCHCLI_URL URL to a custom awsbatch cli package. (default: None) --custom-cw-dashboard-template-url CUSTOM_CW_DASHBOARD_TEMPLATE_URL URL to a custom cw dashboard template. (default: None) --custom-ami CUSTOM_AMI custom AMI to use for all tests. (default: None) --pre-install PRE_INSTALL