文件列表:

Dockerfile (146, 2023-10-30)
configs/ (0, 2023-10-30)
configs/__pycache__/ (0, 2023-10-30)
configs/__pycache__/local.cpython-39.pyc (281, 2023-10-30)
configs/__pycache__/prod.cpython-39.pyc (362, 2023-10-30)
configs/local.py (138, 2023-10-30)
configs/prod.py (217, 2023-10-30)
locustfile.py (2452, 2023-10-30)
requirements.txt (125, 2023-10-30)
run.py (3448, 2023-10-30)
src/ (0, 2023-10-30)
src/tasks/ (0, 2023-10-30)
src/tasks/__pycache__/ (0, 2023-10-30)
src/tasks/__pycache__/hello_world.cpython-39.pyc (365, 2023-10-30)
src/tasks/hello_world.py (249, 2023-10-30)
src/utils/ (0, 2023-10-30)
src/utils/errors.py (1161, 2023-10-30)
src/utils/queue.py (228, 2023-10-30)
src/utils/workers.py (1533, 2023-10-30)

## Data pipeline for kafka # About The Project This is a example for consuming and processing data from kafka also know as data pipelining. * The implementation take advantages from multiprocessing, multithreading, and asynchronous programming * The project designed for single topic with single or multiple partitions * Each partition have it own isolated process that have multiple threads consuming and processing messages * Mainprocess stays idle for now but i keep it anyway maybe some monitoring features for future ## Multiprocessing * An isolated process for each partition topic have ## Multithreading * Each process have one "MainThread" which consumes messages add this messages to queue's for least busy worker thread ## Asynchronous Programming * Each thread has its own event loop running and scheduling tasks * This approach achieves efficient concurrency with io bound callbacks ## Packages * uvloop ~ fast asyncio event loop * aiokafka ~ asynchronous kafka client # task payload example ```json { "task_name": "hello_world", "task_parameters": {} } ```

近期下载者：

相关文件：

评论：[我要评论] [举报此文件]

收藏者：