BiliCenter
所属分类:数据挖掘/数据仓库
开发工具:Python
文件大小:55KB
下载次数:0
上传日期:2023-02-19 12:17:13
上 传 者:
sh-1993
说明: BiliCenter,新版番剧数据中心兼分布式数据挖掘框架
(BiliCenter, New Version of Fanju Data Center and Distributed Data Mining Framework)
文件列表:
LICENSE (35149, 2023-02-19)
bilibili_api_extend (0, 2023-02-19)
bilibili_api_extend\__init__.py (217, 2023-02-19)
bilibili_api_extend\bangumi.py (1109, 2023-02-19)
bilibili_api_extend\common.py (1358, 2023-02-19)
bilibili_api_extend\monkey.py (408, 2023-02-19)
bilibili_api_extend\utils.py (832, 2023-02-19)
bilicenter_middleware (0, 2023-02-19)
bilicenter_middleware\__init__.py (174, 2023-02-19)
bilicenter_middleware\discovery.py (1107, 2023-02-19)
bilicenter_middleware\event.py (2219, 2023-02-19)
bilicenter_middleware\event2job.py (1298, 2023-02-19)
bilicenter_middleware\exceptions.py (265, 2023-02-19)
bilicenter_middleware\logger.py (3958, 2023-02-19)
bilicenter_middleware\statement4SQL.py (1535, 2023-02-19)
docs (0, 2023-02-19)
docs\BiliCenter.svg (58494, 2023-02-19)
docs\config.md (817, 2023-02-19)
docs\env.md (1330, 2023-02-19)
docs\mod_biliCenter.sql (7577, 2023-02-19)
docs\protocol.md (7630, 2023-02-19)
requirements.txt (91, 2023-02-19)
scf (0, 2023-02-19)
scf\biliHelper (0, 2023-02-19)
scf\biliHelper\job (0, 2023-02-19)
scf\biliHelper\job\beitou.py (968, 2023-02-19)
scf\biliHelper\job\biliCenter.py (834, 2023-02-19)
scf\biliHelper\jobs.py (833, 2023-02-19)
scf\biliHelper\router.py (1051, 2023-02-19)
scf\iper (0, 2023-02-19)
scf\iper\index.py (317, 2023-02-19)
service (0, 2023-02-19)
service\CallbackCenter.py (6540, 2023-02-19)
service\ConcurrentController.py (8841, 2023-02-19)
service\FrontEndTrigger.py (6490, 2023-02-19)
service\callback (0, 2023-02-19)
service\callback\bangumi.py (6781, 2023-02-19)
... ...
# BiliCenter
番剧数据中心
一个以事件驱动的数据挖掘框架
## 部署
### 需求
Python 3.6+
参照如下命令安装依赖
```
pip install -r requirements.txt
```
或者
```
python -m pip install -r requirements.txt
```
如果你运行在Linux环境,请视情况将`pip`和`python`替换为`pip3`和`python3`
### 环境与配置
- 请确保拥有腾讯云SCF使用权限,并将`scf/biliHelper`下的SCF函数部署至腾讯云
- 参照[环境变量文档](https://github.com/TiyaAnlite/BiliCenter/blob/master/docs/env.md)进行环境变量的配置,其中包括数据库相关配置
- 根据提供的[SQL](https://github.com/TiyaAnlite/BiliCenter/blob/master/docs/mod_biliCenter.sql)来创建相关的表
### 首次运行
- 定位至`service`下,运行`ConcurrentController.py`,第一次运行会往Redis中写入初始配置并退出
- 可以参照[配置说明](https://github.com/TiyaAnlite/BiliCenter/blob/master/docs/config.md)按需修改运行时配置
### 正式运行
中间件主体在`service`下的三个程序
- `FrontEndTrigger.py`为前端触发器
- `ConcurrentController.py`并发中心
- `CallbackCenter.py`为回调/数据处理中心
启动这三个中间件(无顺序要求),三个中间件无需在同个节点上,只需向同一个数据库连接即可完成通讯
*执行时的目录为`service/`,不正确的目录会导致错误*
### 任务负载
`service/trigger`和`service/callback`下分别为触发器和回调,可自行新增,中间件启动时会自动进行发现
**本文档还有待进一步完善,关于任务负载的定义、触发规则等细节,移步[协议说明](https://github.com/TiyaAnlite/BiliCenter/blob/master/docs/protocol.md)文档**
## 系统设计
![BiliCenter](https://github.com/TiyaAnlite/BiliCenter/blob/master/docs/BiliCenter.svg)
近期下载者:
相关文件:
收藏者: