schematizer:架构存储服务,用于跟踪和管理数据管道中使用的所有架构

  • H7_277301
    了解作者
  • 2.2MB
    文件大小
  • zip
    文件格式
  • 0
    收藏次数
  • VIP专享
    资源类型
  • 0
    下载次数
  • 2022-04-28 19:21
    上传日期
原理图器 它是什么? Schematizer是一种架构存储服务,可跟踪和管理数据管道中使用的所有架构,并提供自动文档支持等功能。 我们使用Apache Avro表示我们的模式。 如何下载 git clone git@github.com:Yelp/schematizer.git 测验 运行单元测试 make -f Makefile-opensource test 运行单元集成测试 make -f Makefile-opensource itest 设置和配置 为Schematizer服务创建mysql数据库: CREATE DATABASE <db> DEFAULT CHARACTER SET utf8; 在<db>数据库中为Schematizer服务创建MySQL表: cat schema/tables/*.sql | mysql <db> 创建一个t
schematizer-master.zip
内容介绍
# Schematizer What is it? ----------- The Schematizer is a schema store service that tracks and manages all the schemas used in the Data Pipeline and provides features like automatic documentation support. We use Apache Avro to represent our schemas. [Read More](https://engineeringblog.yelp.com/2016/08/more-than-just-a-schema-store.html) How to download --------------- ``` git clone git@github.com:Yelp/schematizer.git ``` Tests ----- Running unit tests ``` make -f Makefile-opensource test ``` Running unit integration tests ``` make -f Makefile-opensource itest ``` Setup and Configuration ----------------------- 1. Create a mysql database for Schematizer Service:: ``` CREATE DATABASE <db_name> DEFAULT CHARACTER SET utf8; ``` 2. Create MySQL tables in `<db_name>` database for Schematizer Service:: ``` cat schema/tables/*.sql | mysql <db_name> ``` 3. Create a `topology.yaml` file ``` topology: - cluster: <schematizer_cluster_name> replica: master entries: - charset: utf8 use_unicode: true host: <db_ip> db: <db_name> user: <db_user> passwd: <db_password> port: <db_port> ``` 4. In `config.yaml` assign values to the following configs:: ``` schematizer_cluster: <schematizer_cluster_name> topology_path: /path/to/topology.yaml ``` Usage ----- Use `serviceinitd/schematizer.py` to start the Schematizer service. ### Interactive directly with Schematizer Service. Registering a schema:: ``` curl -X POST --header 'Content-Type: application/json' --header 'Accept: text/plain' -d '{ "namespace": "test_namespace", "source_owner_email": "test@test.com", "source": "test_source", "contains_pii": false, "schema": "{\"type\":\"record\",\"namespace\":\"test_namespace\",\"source\":\"test_source\",\"name\":\"test_name\",\"doc\":\"test_doc\",\"fields\":[{\"type\":\"string\",\"doc\":\"test_doc1\",\"name\":\"key1\"},{\"type\":\"string\",\"doc\":\"test_doc2\",\"name\":\"key2\"}]}" }' 'http://127.0.0.1:8888/v1/schemas/avro' ``` Getting Schema By ID:: ``` curl -X GET --header 'Accept: text/plain' 'http://127.0.0.1:8888/v1/schemas/<schema_id>' ``` ### Interactive with Schematizer Service using Schematizer Client Lib. Registering a schema:: ``` from data_pipeline.schematizer_clientlib.schematizer import get_schematizer test_avro_schema_json = { "type": "record", "namespace": "test_namespace", "source": "test_source", "name": "test_name", "doc": "test_doc", "fields": [ {"type": "string", "doc": "test_doc1", "name": "key1"}, {"type": "string", "doc": "test_doc2", "name": "key2"} ] } schema_info = get_schematizer().register_schema_from_schema_json( namespace="test_namespace", source="test_source", schema_json=test_avro_schema_json, source_owner_email="test@test.com", contains_pii=False ) ``` Getting Schema By ID:: ``` from data_pipeline.schematizer_clientlib.schematizer import get_schematizer schema_info = get_schematizer().get_schema_by_id( schema_id=schema_info.schema_id ) ``` Disclaimer ------- We're still in the process of setting up this service as a stand-alone. There may be additional work required to run a Schematizer instance and integrate with other applications. License ------- Schematizer is licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0 Contributing ------------ Everyone is encouraged to contribute to Schematizer by forking the Github repository and making a pull request or opening an issue.
评论
    相关推荐
    • H2Database内存数据库
      H2Database内存数据库jar包及示例.
    • databasedemo
      databasedemo
    • H2 Database With Docs
      H2 Database With Docs
    • testDatabase
      testDatabase
    • kih_database
      kih_database 基于的国家系统,用于收集、存储和提供跨系统和部门的远程医疗测量和监测数据。 请注意 此存储库由 Silverbullet 独家维护。 4S维护的官方版本可以在找到。
    • FitnessApp-Database
      Fitness App-数据库 ... 为了可以调用该网站, htdocs整个内容htdocs到htdocs文件夹中就... 然后可以在XAMPP中启动Web服务器或Apache。 现在应该可以通过访问该网站。 登录数据也可以在MySQL_Commands.sql文件中找到。
    • Apache HBase Primer
      Learn the fundamental foundations and concepts of the Apache HBase (NoSQL) open source database. It covers the HBase data model, architecture, schema design, API, and administration.Apache HBase is ...
    • Apache Shiro源码
      Apache Shiro 源码 ,Apache Shiro 身份认证例子-源码 .
    • apache 参考手册
      apache 参考手册 apache 参考手册 apache 参考手册
    • apache-jmeter安装
      apache-jmeter安装,apache-jmeter-5.1.1安装包