deep_nowcaster:DFW IPW +降水预测算法

  • I8_334834
    了解作者
  • 106.2MB
    文件大小
  • zip
    文件格式
  • 0
    收藏次数
  • VIP专享
    资源类型
  • 0
    下载次数
  • 2022-05-16 11:22
    上传日期
deep_nowcaster 介绍 该存储库包含再现我的硕士论文探索结果所需的所有代码。 作为副产品,此存储库还包含用于从FTP数据库访问GPS RINEX文件和来自NCDC的ASOS数据的API。 我们还开放了脚本以将天气变量映射到GPS站。 依存关系 麻木 科学的 净CDF scikit学习 uda 茶野 千层面 建立训练和测试数据集 第一步是建立不断变化的降水场和不断变化的湿度场(称为NIPW-归一化综合可沉淀水)的训练/测试数据集,每一个都是一个100 x 100的矩阵,在给定的时间步长t下存储为numpy数组。 在运行脚本之前,请确保使用的链接下载所需的数据(约1.7 GB未压缩)以创建这些字段。 该文件包含来自NetDCF格式的NCDC的原始雷达数据以及IPW(2014、2015和2016年来自KFWS周围44个GPS站的综合可沉淀水汽)的点测量。 以下脚本在我们的实验中绘
deep_nowcaster-master.zip
  • deep_nowcaster-master
  • includes
  • nowcast.py
    9.3KB
  • DFWnet.py
    3.8KB
  • PCA_CCA.py
    5.3KB
  • DCNN_network.py
    13.9KB
  • test_predictions.mat
    25.2KB
  • BuildDataSet.py
    38KB
  • ModelMetrics.py
    2.8KB
  • RandomForest_code
  • BuildNowcasterV4.py
    5.3KB
  • RF_CCA_prediction_experiments.py
    10.3KB
  • BuildNowcaster.py
    4KB
  • RF_experiment_2.sh
    2.9KB
  • BuildNowcasterV2.py
    6KB
  • RF_random_points_avg.py
    2.6KB
  • NB_random_points_avg_experiment_2.py
    2KB
  • analyze_data.py
    3.8KB
  • RF_experiment_2_refl.sh
    2.7KB
  • predict_using_ipw_test.py
    2.2KB
  • BuildNowcasterV5.py
    5.2KB
  • make_predictions_using_averages_random_forest.py
    2.1KB
  • BuildNowcasterV3.py
    4.9KB
  • prediction_regimes.py
    5.9KB
  • RF_experiment_3.sh
    3.3KB
  • make_Predictions.py
    2.2KB
  • RF_random_points_avg_experiment_2.py
    2.1KB
  • RF_experiment.sh
    766B
  • RF_experiment_4.txt
    1.6KB
  • plot_results.py
    17.3KB
  • BuildNowcasterV1.py
    6.4KB
  • RF_prediction_experiments.py
    12.4KB
  • RF_experiment_4.sh
    1.6KB
  • make_predictions_using_averages.py
    2KB
  • code
  • .ipynb_checkpoints
  • NEXRAD_S3_Hook-checkpoint.ipynb
    15.4KB
  • explore_data-checkpoint.ipynb
    387.5KB
  • MakeRadarDataSet.py
    5.7KB
  • plot_metrics.py
    2KB
  • IPW_dump_2.py
    3.5KB
  • save_data.py
    2.6KB
  • NN_Nowcaster.py
    6.2KB
  • run_gamit_2.py
    595B
  • build_data_batches.py
    4.8KB
  • initial_analysis.py
    7.1KB
  • thesis_plots.py
    4.4KB
  • plot_filters.py
    1.4KB
  • WXstations_KFWS.py
    2.7KB
  • NEXRAD_GPS_2.py
    3.6KB
  • GetWeatherAnomolies.py
    11.7KB
  • field_averages_random_points.py
    2.1KB
  • convert_to_image.py
    2.5KB
  • plot_metrics_f1.py
    2.8KB
  • NEXRAD_from_S3.py
    1.7KB
  • practice.py
    494B
  • toolsUI-4.6.jar
    48.7MB
  • 2014_storm_dates.py
    1.6KB
  • GPS_IPW_analysis.py
    4.6KB
  • NEXRAD_GPS_NOAA_SOPAC.py
    4.4KB
  • nowcaster_experiments.sh
    149B
  • ReflectivityDataAnalysis.py
    159B
  • plot_results.py
    3KB
  • plot_reflectivity_histogram.py
    3.7KB
  • find_rinex_files.py
    2.1KB
  • prediction_movies.py
    10.2KB
  • reflectivity_plots.py
    7.9KB
  • IPW_dump_1.py
    2.5KB
  • NEXRAD_GPS_1.py
    3.5KB
  • run_metutil.py
    2.1KB
  • verification_code
  • synthesise_data_set_plotsR.py
    2.9KB
  • Validation.py
    2.1KB
  • synthesise_data_set.py
    3KB
  • Validation_1.py
    3.2KB
  • verify_data_set_R.py
    3.7KB
  • verify_data_set_test_case.py
    5.5KB
  • verify_files.py
    1.8KB
  • Preprocessing_code
  • reflectivity_level3_dataset.py
    13.4KB
  • mine_ASOS2.py
    9.7KB
  • reflectivity_ipw_movies.py
    27.7KB
  • multiprocessing_interpolation.py
    5.2KB
  • toolsUI-4.6.jar
    48.7MB
  • asos-stations.csv
    117.8KB
  • asos2metRINEXpipe.py
    7.9KB
  • Plot_43.png
    81.3KB
  • mine_ASOS.py
    3.9KB
  • reflectivity_ipw_movies_gifs.py
    27.7KB
  • missing_value_imputation.py
    5.5KB
  • weights.npy
    3.4MB
  • multi_processor_weights.npy
    3.4MB
  • storm_cases
  • StormCase20150509.ai
    1.5MB
  • 20140829.ai
    1.6MB
  • 20140508.ai
    1.6MB
  • 20150517.ai
    1.6MB
  • 20140717.ai
    1.6MB
  • 20150617.ai
    1.5MB
  • 20150520.ai
    1.5MB
内容介绍
# deep_nowcaster ## Introduction This repository contains all code required to reproduce results from my Masters Thesis Exploration into [machine learning techniques for precipitation nowcasting](https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1501&context=masters_theses_2). As a byproduct this repo also contains APIs to access GPS RINEX files from FTP database, and ASOS data from NCDC. We also open source the script to map weather variables to GPS stations. ## Dependencies 1. numpy 2. scipy 3. netCDF 4. scikit-learn 5. cuda 5. theano 6. lasagne ## Build Training and Test dataset The first step is to build the train/test data set of evolving precipitation fields and evolving moisture fields (termed as NIPW - Normalized Integrated Precipitable Water) each of which is a 100 x 100 matrix stored as a numpy array for a given timestep t. Before running the script please ensure you download the data(~1.7 GB uncompressed) required to make these fields using the from the link [here](http://emmy9.casa.umass.edu/gpsmet/deep_nowcaster/). This file contains the raw radar data from NCDC in NetCDF format and the point measurements of IPW (Integrated Precipitable Water Vapor from the 44 GPS stations around KFWS for the years 2014,2015 and 2016). The following script plots all the images for the storm dates in our experiment and also stores the images in a numpy array inside data/dataset/YYYY. From inside the Preprocessing_code directory run: ```python python reflectivity_ipw_movies.py ``` 48 plots (NIPW and reflectivity sampled at 30 minute intervals) and numpy arrays for each day are generated. The following shows example plots of the precipitation fields overlapped over the NIPW fields. The video sequence of the evolving precipitation fields and NIPW fields can be found in this [youtube video](https://www.youtube.com/watch?v=r_LATx7BdUQ). ![example plot](https://github.com/adityanagara/deep_nowcaster/blob/master/Preprocessing_code/Plot_43.png) We explore machine learning techniques which can capture the spatiotemporal relationships between the evolving precipitation fields and NIPW fields to be able to nowcast precipitation. ## Train and Test models [BuildDataSet.py](https://github.com/adityanagara/deep_nowcaster/blob/master/includes/BuildDataSet.py) and other scripts in the includes directory contains helper functions to build training and validation data sets and calculating performance metrics, ensure that includes directory is added to your `PYTHON_PATH` variables. ### Random Forest(RF) Classifier We train a random forest classifier using a set of features engineered by taking the spatial statistics of the 33 x 33 window of points around the pixel point we are predicting. The set of routines in [BuildDataSet.py](https://github.com/adityanagara/deep_nowcaster/blob/master/includes/BuildDataSet.py) does this for us and creates a dataset ready to be trained by the Random Forest. From inside the RandomForest_code directory, run the following script: ```python python RF_prediction_experiments.py True 60 ipw_refl 600 RF_60prediction_ipw_refl_experiment 6 ``` which trains a RF classifier with 600 trees in the forest and a max depth of 6 and saves the results in the file `600RF_60prediction_refl_experiment_max_depth6.pkl`. The file contains all the performance metrics evaluated in the training and validation set as defined by the class `NOWCAST_performance()` in [ModelMetrics.py](https://github.com/adityanagara/deep_nowcaster/blob/master/includes/ModelMetrics.py). ### Convolutional Neural Networks(CNN) Unlike the Random Forest classifier we feed the CNN with the actual 33 x 33 frames around the pixel point as features. The weights of convolution filters are learnt for each variable at each time step. The following script runs a single layer CNN with separate connections for the precipitation fields and separate connections for the NIPW fields. From inside the `CNN_code` directory run: ```python python Deep_NN_nowcasting_experiments.py ``` The program first creates the training and validation dataset inside the directory `data/TrainTest/points/`. We train our CNN using a Tesla K80 GPU on [MGHPCC](http://www.mghpcc.org/).
评论
    相关推荐
    • 高光谱遥感图像数据集
      常用的高光谱遥感图像数据集,包含Indians Pines、Botswana、KSC、PaviaU、Salinas及它们的ground truth矩阵。文件后缀为.mat
    • KITTI数据集点云地图构建资料.rar
      本文描述了如何通过KITTI数据集,读取激光雷达点云数据,并通过ground truth进行点云建图的过程。其代码的主要功能包括:1)点云文件的格式转换2)点云转换矩阵计算3)点云地图构建
    • pmf:MovieLens 100K上的概率矩阵分解
      在此项目中,我们使用MovieLens 100K数据集。 该数据集包含来自943位用户的1,682部电影的100,000个评分。 在此项目中,RMSE(均方根误差)用作度量。 我测试了2种不同的数据分割:密集和稀疏。 数据是随机拆分的,...
    • circuitdata:数据集
      连接组学数据集 此存档的目的是提供各种“计算系统”数据集的规范、清理、正确规范化的版本。 所有文件均以 SQLite 格式提供。 这是因为在处理这些数据集时很难同时获得干净的数据和元数据——亲爱的科学家们,请...
    • honst:根据您的规则修复您的数据集
      在编辑连接的矩阵数据时,需要保持数据完整性。 保持本地状态正确。 演示版 您可以在CodeSandbox上尽情玩乐: 概述 const data = [ { username : "johndoe" , name : "John" , surname : "Doe" , age : "22" } , ...
    • LFW-a 数据集mat文件.zip
      LFW-a 数据集子mat文件,已经做cropped处理切割人脸部分,并且resize成了30x15的大小。只包含样本个数大于等于20的类别。fea为样本矩阵,每行一个样本,gnd为类别标签。
    • 数据结构代码合集
      内含二叉树的遍历、哈夫曼编码、后缀表达式的实现、图的创建与遍历、图的应用、稀疏矩阵、线性表合并、一元多项式的乘法、加法、减法。
    • matrixTests:R包,用于在矩阵数据框的行列上计算多个假设检验
      使用“物种”作为组,对虹膜数据集的每一列进行Bartlett检验: col_bartlett( iris [, - 5 ], iris $ Species ) obs.tot obs.groups var.pooled df statistic pvalue Sepal.Length 150 3 0.26500816 2 16.005702 0...
    • ruse.js:为绘制大型数据集而构建的轻量级 WebGL 库
      ruse.js是一个轻量级的 WebGL 库,用于绘制大型数据集。 它能够一次绘制数百万个点,并在图之间平滑过渡。 依赖 ruse.js仅依赖于进行矩阵转换。 用法 使用常用的脚本标签将gl-matrix和ruse.js包含到项目中。 ...
    • 矩阵分解数据集
      矩阵分解所需数据集,里面数据集有6个,ratings.csv,