BiliCenter

所属分类:数据挖掘/数据仓库
开发工具:Python
文件大小:55KB
下载次数:0
上传日期:2023-02-19 12:17:13
上 传 者sh-1993
说明:  BiliCenter,新版番剧数据中心兼分布式数据挖掘框架
(BiliCenter, New Version of Fanju Data Center and Distributed Data Mining Framework)

文件列表:
LICENSE (35149, 2023-02-19)
bilibili_api_extend (0, 2023-02-19)
bilibili_api_extend\__init__.py (217, 2023-02-19)
bilibili_api_extend\bangumi.py (1109, 2023-02-19)
bilibili_api_extend\common.py (1358, 2023-02-19)
bilibili_api_extend\monkey.py (408, 2023-02-19)
bilibili_api_extend\utils.py (832, 2023-02-19)
bilicenter_middleware (0, 2023-02-19)
bilicenter_middleware\__init__.py (174, 2023-02-19)
bilicenter_middleware\discovery.py (1107, 2023-02-19)
bilicenter_middleware\event.py (2219, 2023-02-19)
bilicenter_middleware\event2job.py (1298, 2023-02-19)
bilicenter_middleware\exceptions.py (265, 2023-02-19)
bilicenter_middleware\logger.py (3958, 2023-02-19)
bilicenter_middleware\statement4SQL.py (1535, 2023-02-19)
docs (0, 2023-02-19)
docs\BiliCenter.svg (58494, 2023-02-19)
docs\config.md (817, 2023-02-19)
docs\env.md (1330, 2023-02-19)
docs\mod_biliCenter.sql (7577, 2023-02-19)
docs\protocol.md (7630, 2023-02-19)
requirements.txt (91, 2023-02-19)
scf (0, 2023-02-19)
scf\biliHelper (0, 2023-02-19)
scf\biliHelper\job (0, 2023-02-19)
scf\biliHelper\job\beitou.py (968, 2023-02-19)
scf\biliHelper\job\biliCenter.py (834, 2023-02-19)
scf\biliHelper\jobs.py (833, 2023-02-19)
scf\biliHelper\router.py (1051, 2023-02-19)
scf\iper (0, 2023-02-19)
scf\iper\index.py (317, 2023-02-19)
service (0, 2023-02-19)
service\CallbackCenter.py (6540, 2023-02-19)
service\ConcurrentController.py (8841, 2023-02-19)
service\FrontEndTrigger.py (6490, 2023-02-19)
service\callback (0, 2023-02-19)
service\callback\bangumi.py (6781, 2023-02-19)
... ...

# BiliCenter 番剧数据中心 一个以事件驱动的数据挖掘框架 ## 部署 ### 需求 Python 3.6+ 参照如下命令安装依赖 ``` pip install -r requirements.txt ``` 或者 ``` python -m pip install -r requirements.txt ``` 如果你运行在Linux环境,请视情况将`pip`和`python`替换为`pip3`和`python3` ### 环境与配置 - 请确保拥有腾讯云SCF使用权限,并将`scf/biliHelper`下的SCF函数部署至腾讯云 - 参照[环境变量文档](https://github.com/TiyaAnlite/BiliCenter/blob/master/docs/env.md)进行环境变量的配置,其中包括数据库相关配置 - 根据提供的[SQL](https://github.com/TiyaAnlite/BiliCenter/blob/master/docs/mod_biliCenter.sql)来创建相关的表 ### 首次运行 - 定位至`service`下,运行`ConcurrentController.py`,第一次运行会往Redis中写入初始配置并退出 - 可以参照[配置说明](https://github.com/TiyaAnlite/BiliCenter/blob/master/docs/config.md)按需修改运行时配置 ### 正式运行 中间件主体在`service`下的三个程序 - `FrontEndTrigger.py`为前端触发器 - `ConcurrentController.py`并发中心 - `CallbackCenter.py`为回调/数据处理中心 启动这三个中间件(无顺序要求),三个中间件无需在同个节点上,只需向同一个数据库连接即可完成通讯 *执行时的目录为`service/`,不正确的目录会导致错误* ### 任务负载 `service/trigger`和`service/callback`下分别为触发器和回调,可自行新增,中间件启动时会自动进行发现 **本文档还有待进一步完善,关于任务负载的定义、触发规则等细节,移步[协议说明](https://github.com/TiyaAnlite/BiliCenter/blob/master/docs/protocol.md)文档** ## 系统设计 ![BiliCenter](https://github.com/TiyaAnlite/BiliCenter/blob/master/docs/BiliCenter.svg)

近期下载者

相关文件


收藏者