monster
所属分类:大数据
开发工具:Shell
文件大小:0KB
下载次数:0
上传日期:2021-07-06 16:50:58
上 传 者:
sh-1993
说明: DSP数据工程中Monster团队的中心
(Hub for the Monster team in DSP Data Engineering)
文件列表:
archive/ (0, 2021-07-06)
getting-started/ (0, 2021-07-06)
getting-started/monster.png (68243, 2021-07-06)
getting-started/pages/ (0, 2021-07-06)
getting-started/pages/dev-setup.md (4638, 2021-07-06)
getting-started/pages/group-team-setup.md (4949, 2021-07-06)
getting-started/pages/new-broadies.md (2197, 2021-07-06)
getting-started/pages/reading-list.md (1089, 2021-07-06)
getting-started/pages/toolbox-icon.png (1453, 2021-07-06)
getting-started/pages/vault-github-token.png (271385, 2021-07-06)
getting-started/scripts/ (0, 2021-07-06)
getting-started/scripts/clone-repositories (1209, 2021-07-06)
getting-started/scripts/install-tools (4349, 2021-07-06)
getting-started/scripts/login (532, 2021-07-06)
getting-started/scripts/setup-graal (1283, 2021-07-06)
getting-started/scripts/setup-vault (1227, 2021-07-06)
tech-docs/ (0, 2021-07-06)
tech-docs/devops/ (0, 2021-07-06)
tech-docs/devops/motivation.md (997, 2021-07-06)
tech-docs/devops/requirements.md (1914, 2021-07-06)
tech-docs/devops/strategy.md (7326, 2021-07-06)
tech-docs/stack/ (0, 2021-07-06)
tech-docs/stack/languages.md (6528, 2021-07-06)
tech-docs/stack/systems.md (8871, 2021-07-06)
tech-docs/stack/tools.md (4498, 2021-07-06)
templates/ (0, 2021-07-06)
templates/python-project/ (0, 2021-07-06)
templates/python-project/.pre-commit-config.yaml (102, 2021-07-06)
... ...
# The Monster Team
[![Monster Slack](https://img.shields.io/badge/Slack%20Channel-%23monster-blue.svg?style=flat)](https://broadinstitute.slack.com/messages/CCAU5L6LV/)
[![Monster CI Slack](https://img.shields.io/badge/Slack%20Channel-%23monster--ci-blue.svg?style=flat)](https://broadinstitute.slack.com/messages/CFXEDUUP5/)
New to the team? [Start here](./getting-started/README.md).
## People
| Name | Role | GitHub |
| --- | --- | --- |
| Jeff Korte | Product Owner | @JeffKorte |
| Quazi Hoque | Software Engineer | @quazi-broad |
| Drew Herbst | Tech Lead | @aherbst-broad |
## GitHub Teams
* [DSP Monsters](https://github.com/orgs/broadinstitute/teams/dsp-monsters) - Team for repositories under the `broadinstitute` org
* [Monster](https://github.com/orgs/DataBiosphere/teams/monster) - Team for repositories under the `DataBiosphere` org
## Projects
### Data Modeling
Linked Data definitions for the Terra Core Data Model, with extensions for unmodeled datasets.
#### Documentation
* [Google Docs](https://drive.google.com/drive/folders/1n8TP4Q_4n2pCysjQz2Hkn2kpHGEILLCj)
* [Confluence - DSP Core Data Model](https://broadinstitute.atlassian.net/wiki/spaces/DSPCDM/overview)
* [Confluence - FAIR Community of Practice](https://broadinstitute.atlassian.net/wiki/spaces/FairCoP/overview)
#### GitHub repos
* [TerraCore Data Model](https://github.com/DataBiosphere/terra-core-data-model) - Data Model definitions and examples
### Data Ingest
Pipelines for moving data into the [Jade Data Repository](https://github.com/databiosphere/jade-data-repo).
#### Documentation
* [Google Docs](https://drive.google.com/drive/folders/1LjtBbMZs5-FqTGcRjw80ZBlHhfd_LT2z)
#### GitHub repos
* [ClinVar](https://github.com/DataBiosphere/clinvar-ingest) - ETL pipeline for the ClinVar dataset
* [ENCODE](https://github.com/DataBiosphere/encode-ingest) - ETL pipeline for the ENCODE dataset
* [Dog Aging](https://github.com/DataBiosphere/dog-aging-ingest) - ETL pipeline for the Dog Aging Project dataset
* [HCA](https://github.com/DataBiosphere/hca-ingest) - ETL pipeline for the HCA
### Ingest Utilities
Tools and libraries used to support the top-level ingest pipelines.
#### GitHub repos
* [Base utilities](https://github.com/DataBiosphere/ingest-utils) - Common utilities shared across our batch ETL projects
* [XML-to-JSON-list](https://github.com/broadinstitute/monster-xml-to-json-list) - Command-line tool for mechanical
conversion of XML into Beam-friendly JSON
### Operations
Infrastructure, configuration, and shared code used to manage developing and deploying our services.
### GitHub repos
* [Helm charts](https://github.com/broadinstitute/monster-helm) - Custom Helm charts for pieces of Monster infrastructure
* [Core deployments](https://github.com/broadinstitute/monster-deploy) - Terraform modules, Helm releases, and deploy scripts
for Monster's GCP environments
* [setup-chart-releaser](https://github.com/broadinstitute/setup-chart-releaser) - GitHub Action to install [Chart Releaser](https://github.com/helm/chart-releaser)
## Semi-Archived
The repositories in this section are still being used, but we're trying to move away from them.
### Data Ingest Framework
Our first stabs at data ingest envisioned a framework of dataset-agnostic services.
We shifted away from that pattern because it introduced significant overhead vs. custom
pipelines using common command-line tools.
#### GitHub repos
* [Transporter](https://github.com/databiosphere/transporter) - Bulk file-transfer system
* [Storage Libs](https://github.com/broadinstitute/monster-storage-libs) - Utility libraries for I/O against external storage systems
近期下载者:
相关文件:
收藏者: