文件列表:

.vscode/ (0, 2021-07-30)
.vscode/desktop.ini (246, 2021-07-30)
.vscode/settings.json (92, 2021-07-30)
Dockerfile (184, 2021-07-30)
Report of the project.pdf (1342319, 2021-07-30)
Shiny Bolzano Application/ (0, 2021-07-30)
Shiny Bolzano Application/.RData (66365, 2021-07-30)
Shiny Bolzano Application/.Rhistory (15094, 2021-07-30)
Shiny Bolzano Application/Pipeline.png (1375580, 2021-07-30)
Shiny Bolzano Application/Traffic-Bolzano.Rmd (16625, 2021-07-30)
Shiny Bolzano Application/Traffic.Rproj (205, 2021-07-30)
Shiny Bolzano Application/db_connection.R (529, 2021-07-30)
Shiny Bolzano Application/desktop.ini (246, 2021-07-30)
Shiny Bolzano Application/rsconnect/ (0, 2021-07-30)
Shiny Bolzano Application/rsconnect/documents/ (0, 2021-07-30)
Shiny Bolzano Application/rsconnect/documents/Traffic-Bolzano.Rmd/ (0, 2021-07-30)
Shiny Bolzano Application/rsconnect/documents/Traffic-Bolzano.Rmd/bookdown.org/ (0, 2021-07-30)
Shiny Bolzano Application/rsconnect/documents/Traffic-Bolzano.Rmd/bookdown.org/aurora_tumminello/ (0, 2021-07-30)
Shiny Bolzano Application/rsconnect/documents/Traffic-Bolzano.Rmd/bookdown.org/aurora_tumminello/traffic-bolzano.dcf (335, 2021-07-30)
Shiny Bolzano Application/rsconnect/documents/Traffic-Bolzano.Rmd/shinyapps.io/ (0, 2021-07-30)
Shiny Bolzano Application/rsconnect/documents/Traffic-Bolzano.Rmd/shinyapps.io/traffic-bolzen/ (0, 2021-07-30)
Shiny Bolzano Application/rsconnect/documents/Traffic-Bolzano.Rmd/shinyapps.io/traffic-bolzen/traffic-bolzano2.dcf (316, 2021-07-30)
Shiny Bolzano Application/styles.css (345, 2021-07-30)
__pycache__/ (0, 2021-07-30)
__pycache__/desktop.ini (246, 2021-07-30)
data/ (0, 2021-07-30)
data/desktop.ini (246, 2021-07-30)
data/latest_data.csv (24, 2021-07-30)
data/model/ (0, 2021-07-30)
data/model/model.h5 (63984, 2021-07-30)
data/model/model.json (1575, 2021-07-30)
db/ (0, 2021-07-30)
db/.DS_Store (6148, 2021-07-30)
... ...

# Traffic-Project ## Our aim The purpose of our project was to design and implement a big data system able to predict in near-time the traffic associated with specific and crucial points of interest monitored by Bluetooth sensors in the Bolzano area, using Open Data Hub Südtirol databases. ## Basic structure of the repository Sections: 1. Code: all folders and files involved 2. Action: there are two different workflows, the first one (i.e real time data collection)regards the download_real_time script while the second one (i.e model prediction)regards the loop prediction script. 3. Packages: "bdt_new" is a package cointaining docker images of RDS MySQL database, Python and R files. ## How to execute specific stages of the pipeline ### Data collection Data collection is handled by two different scripts: - `download_history`, locally executed in order to get data in batch; - `download_real_time`, whose execution is driven by Github Actions by installing requirements and showing some logs inside the specific build (in particular, inside the step "execute script"). It is scheduled to be executed every 30 minutes, since publishers release data every 10 minutes. All requirements for both scripts are contained inside `py/download_requirements.txt`. `BluetoothStation` script contains three classes that describe three abstract objects: Bluetooth Station, with its geographical coordinates, described by the class Position; Measurement, which considers timestamp, Bluetooth Station and count of vehicles passing. ### Data ingestion `DatabaseManager` script instead offers a variety of database managers, depending on the provider. The most basic one is a local MySQL database, the following one is still a MySQL database, but hosted on Amazon RDS (remote connection), whose connection is available through secret credentials. In the end, a simple MongoDB manager was created to insert and delete many observations in a row inside the collection. Inside the `db` folder, it is possible to find several scripts for emulating the construction of MySQL database: - `bluetooothstations.sql` contains the creation schema of all tables inside the db; - `time_group.sql` and `traffic_per_hour.sql`describe the creation of two summary tables used to query efficiently the database inside the application. - all the triggers specified are necessary to update sum and averages inside the previous two tables whenever new observations are inserted in the Database. ### Model `Traffic-Bolzano.ipynb` specific notebook for the training phase of the model. In order to run it, it is necessary to use [Databricks](https://databricks.com/), that exploits Spark to handle the massive amount of data for the training phase. `Traffic_Loop.py` script aims to continuosly predict the traffic for every station, considering the latest two hours of data at the start and then concatenating results till they get to the current timestamp. The entire workflow is handled through GitHub Actions. Requirements for this script are contained inside the `py/model_requirements.py`. Model structure and weights are saved inside `data/model` directory. They were initially exported from the training phase in json and h5 format and then imported inside the loop and continuously updated. ### Web Application Inside `Shiny Bolzano Application`, it is possibile to find the original scripts, for both configuration and running, used to build the web application hosted on Shiny Server at the following [link](https://traffic-bolzano.shinyapps.io/traffic-bolzano2/). `db_connection.R` is the script that contains two functions to connect the application directly to MySQL and MongoDB databases, whereas `Traffic-Bolzano.Rmd` is the actual application written in RMarkdown with the help of shiny and flexdashboard.

近期下载者：

相关文件：

评论：[我要评论] [举报此文件]

收藏者：