chaperone:Kafka审核系统

  • h0_351683
    了解作者
  • 238.6KB
    文件大小
  • zip
    文件格式
  • 0
    收藏次数
  • VIP专享
    资源类型
  • 0
    下载次数
  • 2022-06-05 10:42
    上传日期
伴侣分子 (不建议使用此项目,并且不对其进行维护。) 作为Kafka审核系统,伴侣可以监视数据流的完整性和延迟。 Kafka用户的审核指标将保留在数据库中,以量化其主题的损失(如果有)。 基本上,伴侣将时间线切成10分钟的存储桶,并根据其事件时间将消息分配给相应的存储桶。 存储桶的统计信息会相应更新,例如消息总数。 这些统计数据会定期发送给专门的Kafka主题,例如“陪伴审核”。 ChaperoneCollector将使用本主题中的那些统计信息并将其持久保存到数据库中。 ============伴侣分子由几种成分组成: ChaperoneClient是一个库,可以像Kafka Producer或Consumer一样放入库中,以在消息通过时对其进行审核。 审核统计信息将发送到专门的Kafka主题,例如“陪伴审核”。 ChaperoneCollector消耗“ chaperone-a
chaperone-master.zip
内容介绍
Chaperone ============ (This project is deprecated and not maintained.) As Kafka audit system, Chaperone monitors the completeness and latency of data stream. The audit metrics are persisted in database for Kafka users to quantify the loss of their topics if any. Basically, Chaperone cuts timeline into 10min buckets and assigns message to corresponding bucket according to its event time. The stats of the bucket are updated accordingly, like the total message count. Periodically, the stats are sent out to a dedicated Kafka topic, say 'chaperone-audit'. ChaperoneCollector consumes those stats from this topic and persists them into database. ============ Chaperone is made of several components: 1. ChaperoneClient is a library that can be put in like Kafka Producer or Consumer to audit messages as they flow through. The audit stats are sent to a dedicated Kafka topic, say 'chaperone-audit'. 2. ChaperoneCollector consumes audit stats from 'chaperone-audit' and persists them into database. 3. ChaperoneService audits messages kept in Kafka. Since it's built upon [uReplicator](https://github.com/uber/uReplicator), it consists of two subsystems: ChaperoneServiceController to auto-detect topics in Kafka and assign the topic-partitions to workers to audit; ChaperoneServiceWorker to audit messages from assigned topic-partitions. In particular, ChaperoneService and ChaperoneCollector together ensure each message is audited exactly once. # Chaperone Quick Start ## Get the Code Check out the Chaperone project: ``` git clone git@github.com:uber/chaperone.git cd chaperone ``` This project contains everything you’ll need to run Chaperone. ## Build Chaperone Before you can run Chaperone, you need to build a package for it. ``` mvn clean package ``` Or command below to skip tests ``` mvn clean package -DskipTests ``` ## Set Up Local Test Environment To test Chaperone locally, you need two systems: [Kafka](http://kafka.apache.org/), and [ZooKeeper](http://zookeeper.apache.org/). The script “grid” is to help you set up these systems. - The command below will download, install, and start ZooKeeper and Kafka (named cluster1) ``` bin/grid bootstrap ``` ## Start ChaperoneService - Start ChaperoneService Controller ``` ./ChaperoneDistribution/target/ChaperoneDistribution-pkg/bin/start-chaperone-controller.sh ``` - Start ChaperoneService Worker ``` ./ChaperoneDistribution/target/ChaperoneDistribution-pkg/bin/start-chaperone-worker.sh ``` ## Generate Load - Create a dummyTopic in Kafka and produce some dummy data: ``` ./bin/produce-data-to-kafka-topic-dummyTopic.sh ``` - Check if the data is successfully produced to Kafka by console-consumer as below: ``` ./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181/cluster1 --topic dummyTopic ``` You should get this data: ``` Kafka topic dummy topic data 1 Kafka topic dummy topic data 2 Kafka topic dummy topic data 3 Kafka topic dummy topic data 4 … ``` ## Check Audit Stats In this example, the topic dummyTopic will be auto-detected and assigned to worker to audit. Periodically, the audit stats are sent to a topic called 'chaperone-audit'. ``` ./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181/cluster1 --topic chaperone-audit ``` One can also manually add topic to audit by command below: ``` curl -X POST -d '{"topic":"dummyTopic", "numPartitions":"1"}' http://localhost:9000/topics ``` ## Start ChaperoneCollector To start ChaperoneCollector, MySQL is required and Redis is optional. MySQL is used to persist audit stats and Redis is used to deduplicate. Deduplication can be turned off. The configuration file for ChaperoneCollector is ./config/chaperonecollector.properties, which might be updated to connect to MySQL and Redis. - Start ChaperoneCollector ``` ./ChaperoneDistribution/target/ChaperoneDistribution-pkg/bin/start-chaperone-collector.sh ```
评论
    相关推荐
    • Kafka技术内幕
      Kafka技术内幕 Kafka技术内幕:图文详解Kafka源码设计与实现.郑奇煌
    • kafka elasticsearch
      通过获取kafka消息列队消费发送到elasticsearch做持久存储
    • Kafka管理工具Kafka Tool
      Kafka Tool是一个用于管理和使用Apache Kafka集群的GUI应用程序。它提供了一个直观的UI,允许用户快速查看Kafka群集中的对象以及存储在群集主题中的消息。
    • Kafka技术内幕
      本书以0.10版本的源码为基础,深入分析了Kafka的设计与实现,包括生产者和消费者的消息处理流程,新旧消费者不同的设计方式,存储层的实现,协调者和控制器如何确保Kafka集群的分布式和容错特性,两种同步集群工具...
    • kafka-playground
      卡夫卡游乐场 一个用于存放一些与Kafka相关的测试代码的运动场存储库。
    • Kafka_Learn.zip
      该代码包含kafka的生产者、消费者原理详解,各种参数解析,主题、分区、存储等的代码演示,可用于搭配博客学习
    • kafka-journal:使用Kafka作为主要存储的事件源日志实现
      kafka-journal:使用Kafka作为主要存储的事件源日志实现
    • zipkin-storage-kafka:基于Kafka的Zipkin存储
      邮编存储Kafka [EXPERIMENTAL] 基于Kafka的Zipkin存储。 +----------------------------*zipkin*---------------------------------------------- | [ dependency-storage ]--->( dependencies ) | ^ +--&...
    • kafka
      该项目需要将ojdbc jar文件添加到本地Maven存储库中 jar文件在仓库中可用 mvn install:安装文件-Dfile = ojdbc6.jar -DgroupId = com.oracle -DartifactId = ojdbc6 -Dversion = 12.1.0.2 -Dpackaging = jar
    • SIM800C_MQTT.rar
      使用SIM800C模块,使用MQTT协议,连接中国移动onenet平台,能实现数据的订阅、发布、存储等