实时自适应深度立体声:“实时自适应深度立体声”的代码-CVPR 2019(ORAL)

  • v7_485766
    了解作者
  • 112.4KB
    文件大小
  • zip
    文件格式
  • 0
    收藏次数
  • VIP专享
    资源类型
  • 0
    下载次数
  • 2022-06-13 04:31
    上传日期
实时自适应深度立体声 刚刚上传到arxive的扩展版本: 。 扩展版本的代码将很快发布。 ,在CVPR 2019上被视为ORAL 。 | | | 摘要: 端到端训练的深度卷积神经网络是无可争议的最新技术,可以直接从立体对中回归密集的视差图。 然而,当这些方法暴露于与训练阶段所见的场景(例如真实图像,合成图像,室内图像与室外图像等)截然不同的场景时,其准确性会明显下降。 由于不可能在任何目标域中收集足够的样本来实现有效的训练/调整,我们建议对深度立体声网络进行无监督且连续的在线适配,以独立于所感测的环境来保持其准确性。 但是,这种策略在计算资源方面可能要求很高,因此无法实现实时性能。
Real-time-self-adaptive-deep-stereo-master.zip
  • Real-time-self-adaptive-deep-stereo-master
  • README.MD
    6.1KB
  • example_list.csv
    95.4KB
  • images
  • MADNet.png
    60.7KB
  • Losses
  • loss_factory.py
    9.9KB
  • __init__.py
    0B
  • Sampler
  • sampler_factory.py
    2.5KB
  • __init__.py
    0B
  • Train.py
    7.6KB
  • Nets
  • __init__.pyc
    672B
  • Stereo_net.py
    7.9KB
  • __init__.py
    360B
  • MadNet.py
    20.4KB
  • DispNet.py
    8.5KB
  • Native
  • shift_corr.cc
    3.5KB
  • shift_corr.cu.cc
    9.3KB
  • Makefile
    2.1KB
  • sharedLayers.py
    7KB
  • Demo
  • grabber.py
    5.4KB
  • Live_Adaptation_Demo.py
    2.8KB
  • demo_model.py
    11.1KB
  • Stereo_Online_Adaptation.py
    12.2KB
  • LICENSE
    11.1KB
  • block_config
  • MadNet_full.json
    1.2KB
  • dispnet_full.json
    438B
  • MadNet_piramid_only.json
    1.1KB
  • Data_utils
  • preprocessing.py
    10.3KB
  • data_reader.py
    6.1KB
  • __init__.py
    0B
  • weights_utils.py
    3KB
  • requirements.txt
    68B
  • .gitignore
    62B
内容介绍
# Real-Time Self-Adaptive Deep Stereo **Extended version just uploaded to arxive: [Link](https://arxiv.org/pdf/2007.05233.pdf). Code for the extended version will be released soon.** Code for [Real-time self-adaptive deep stereo](https://arxiv.org/abs/1810.05424), accepted as **ORAL** at **CVPR 2019**. | [paper](https://arxiv.org/abs/1810.05424) | [video](https://www.youtube.com/watch?v=7SjyzDxmCY4) | [CVPR_Live_Demo](https://www.youtube.com/watch?v=4O-7OzVYAeU) | ![image](images/MADNet.png) **Abstract**: Deep convolutional neural networks trained end-to-end are the undisputed state-of-the-art methods to regress dense disparity maps directly from stereo pairs. However, such methods suffer from notable accuracy drops when exposed to scenarios significantly different from those seen in the training phase (e.g.real vs synthetic images, indoor vs outdoor, etc). As it is unlikely to be able to gather enough samples to achieve effective training/ tuning in any target domain, we propose to perform unsupervised and continuous online adaptation of a deep stereo network in order to preserve its accuracy independently of the sensed environment. However, such a strategy can be extremely demanding regarding computational resources and thus not enabling real-time performance. Therefore, we address this side effect by introducing a new lightweight, yet effective, deep stereo architecture Modularly ADaptive Network (MADNet) and by developing Modular ADaptation (MAD), an algorithm to train independently only sub-portions of our model. By deploying MADNet together with MAD we propose the first ever realtime self-adaptive deep stereo system. If you use this code please cite: ``` @InProceedings{Tonioni_2019_CVPR, author = {Tonioni, Alessio and Tosi, Fabio and Poggi, Matteo and Mattoccia, Stefano and Di Stefano, Luigi}, title = {Real-time self-adaptive deep stereo}, booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2019} } ``` ## Requirements This software has been tested with python3 and tensorflow 1.10. All required packages can be installed using pip and requirements.txt ``` pip3 install -r requirements.txt ``` ## Pretrained Weights for Network Pretrained weights for both DispNet and MADNet available [here](https://drive.google.com/open?id=1GwGxBOFx-NlUo9RAUgPlgPvaHCpGedlm). ## Online Adaptation Step by step on Video Sequences 1. Create a csv file for your video sequence similar to `example_list.csv`. Each row should contain absolute paths to the input data in the following order: "*path_to_left_rgb*,*path_to_right_rgb*,*path_to_groundtruth*" Ground truth data will only be used to compute the network performance not for the online adaptation. 2. Download pretrained network from [here](https://drive.google.com/open?id=1GwGxBOFx-NlUo9RAUgPlgPvaHCpGedlm). 3. Perform different kinds of online adaptation with `Stereo_Online_Adaptation.py`, to list all available options use `python3 Stereo_Online_Adaptation.py -h`. Example of online adaptation using MADNet and MAD: ```bash LIST="path/to/the/list/of/frames/" #the one described at step (1) OUTPUT="path/to/output/folder" WEIGHTS="path/to/pretrained/network" MODELNAME="MADNet" BLOCKCONFIG="block_config/MadNet_full.json" python3 Stereo_Online_Adaptation.py \ -l ${LIST} \ -o ${OUTPUT} \ --weights ${WEIGHTS} \ --modelName MADNet \ --blockConfig ${BLOCKCONFIG} \ --mode MAD \ --sampleMode PROBABILITY \ --logDispStep 1 ``` Additional details on available parameters: + **--modelName**: choose which disparity inference model to use, currently supports either "DispNet" or "MADNet" + **--mode**: how to perform online adaptation: NONE (no online adaptation only inference), FULL (adaptation by full online backprop), MAD (adaptation by partial backprop using MAD) + **--blockConfig**: path to a json file that specify how to subdivide the network into independent blocks for adaptation using MAD, some example are provided inside the `block_config` folder. Each innermost array inside the json file should correspond to layers belonging to the same independent group. + **--sampleMode**: specify how to sample portions of the network to train when using MAD. + **--weights**: path to the initial weights for the network, e.g., referring to the weights linked above and assuming of having extracted the zip file in the same folder as the python script, to load MADNet trained on synthetic data use `--weights MADNet/synthetic/weights.ckpt` + **--logDispStep**: save the disparity predicted every this number of steps. ## Live inference and adaptation demo The Demo sub-folder contains python code to run stereo online inference and/or adaptation from a webcam stream. Before running the demo you should implement in `Demo/grabber.py` a class that extends `ImageGrabber` and is tagged with the `@register_camera_to_factory()` decorator. The class should handle connection and disconnection from your stereo camera as well as all the op necessary to read/decode and rectify stereo frames. The file `grabber.py` contains: a commented `SmattCam` class with an implementation for a custom stereo camera implemented on an FPGA and a `ZEDMini` class to connect to a ZED stereo camera. Once you have implemented the code to connect to your camera you should be able to run the demo using the `Demo/Live_Adaptation_Demo.py` script. The script accepts a bunch of parameters from the command line to, all the options can be listed using ```python python Live_Detection_Demo.py -h ``` Once started the script spawn two threads, one to read frames from the camera the others to process them through a stereo CNN. To stop execution just press something followed by `enter` on the shell where `Live_Detection_Demo.py` is running. ## Native Correlation layers All the test in the paper were conducted using correlation layers implemented in pure tensorflow operations, however we provide also a native CUDA version of the same layer that might be usefull for deployment on memory constrained devices. To use the native correlation layer: 1. `cd Nets/Native` 2. `make` 3. `cd ..` 4. `vim sharedLayers.py` 5. Comment line 6, uncomment line from 12 to 19
评论
    相关推荐