BatchCenterTWMD
所属分类：聚类算法
开发工具：Python
文件大小：28988KB
下载次数：0
上传日期：2020-10-19 05:05:42
上 传 者：
sh-1993
说明： 基于调节词移动距离和批中心词向量的句子间相似性度量
(sentence-to-sentence similarity metric with tempered Word Mover Distance and batch centered word vectors)
文件列表:
LICENSE (11357, 2020-10-19)
batch_center_eval.py (8038, 2020-10-19)
corpus_mean.png (221932, 2020-10-19)
dataset (0, 2020-10-19)
dataset\STS (0, 2020-10-19)
dataset\STS\STS12-en-test (0, 2020-10-19)
dataset\STS\STS12-en-test\STS.gs.ALL.txt (18648, 2020-10-19)
dataset\STS\STS12-en-test\STS.gs.MSRpar.txt (4500, 2020-10-19)
dataset\STS\STS12-en-test\STS.gs.MSRvid.txt (4500, 2020-10-19)
dataset\STS\STS12-en-test\STS.gs.SMTeuroparl.txt (2754, 2020-10-19)
dataset\STS\STS12-en-test\STS.gs.surprise.OnWN.txt (4500, 2020-10-19)
dataset\STS\STS12-en-test\STS.gs.surprise.SMTnews.txt (2394, 2020-10-19)
dataset\STS\STS12-en-test\STS.input.MSRpar.txt (168149, 2020-10-19)
dataset\STS\STS12-en-test\STS.input.MSRvid.txt (50402, 2020-10-19)
dataset\STS\STS12-en-test\STS.input.SMTeuroparl.txt (61015, 2020-10-19)
dataset\STS\STS12-en-test\STS.input.surprise.OnWN.txt (68924, 2020-10-19)
dataset\STS\STS12-en-test\STS.input.surprise.SMTnews.txt (59937, 2020-10-19)
dataset\STS\STS13-en-test (0, 2020-10-19)
dataset\STS\STS13-en-test\STS.gs.FNWN.txt (685, 2020-10-19)
dataset\STS\STS13-en-test\STS.gs.OnWN.txt (2001, 2020-10-19)
dataset\STS\STS13-en-test\STS.gs.SMT.txt (2753, 2020-10-19)
dataset\STS\STS13-en-test\STS.gs.headlines.txt (2716, 2020-10-19)
dataset\STS\STS13-en-test\STS.input.FNWN.txt (46833, 2020-10-19)
dataset\STS\STS13-en-test\STS.input.OnWN.txt (48351, 2020-10-19)
dataset\STS\STS13-en-test\STS.input.headlines.txt (69813, 2020-10-19)
dataset\STS\STS13-en-test\STS.output.FNWN.txt (2816, 2020-10-19)
dataset\STS\STS13-en-test\STS.output.OnWN.txt (8369, 2020-10-19)
dataset\STS\STS13-en-test\STS.output.SMT.txt (11182, 2020-10-19)
dataset\STS\STS13-en-test\STS.output.headlines.txt (11173, 2020-10-19)
dataset\STS\STS13-en-test\correct-output.pl (1465, 2020-10-19)
dataset\STS\STS13-en-test\correlation-all.pl (898, 2020-10-19)
dataset\STS\STS13-en-test\correlation.pl (1894, 2020-10-19)
dataset\STS\STS14-en-test (0, 2020-10-19)
dataset\STS\STS14-en-test\STS.gs.OnWN.txt (2765, 2020-10-19)
dataset\STS\STS14-en-test\STS.gs.deft-forum.txt (1780, 2020-10-19)
... ...
# Improving Text Generation Evaluation with Batch Centering and Tempered Word Mover Distance
Automatic evaluation metrics of Tempered-Word Mover Distance and batch centered word vectors described in [Improving Text Generation Evaluation with Batch Centering and Tempered Word Mover Distance](https://arxiv.org/abs/2010.06150)
### Overview
We present two techniques for improving encoding representations for similarity metrics:
- a **batch-mean centering** strategy that improves statistical properties
- a computationally efficient **tempered Word Mover Distance (TWMD)**, for better fusion of the information in the contextualized word representations.
We conduct numerical experiments that demonstrate the robustness of our techniques, reporting results over various BERTbackbone learned metrics and achieving state of the art correlation with human ratings on several benchmarks.
This repo made use of the [transformers](https://github.com/huggingface/transformers) pacakage.
### Installation
#### Create a new environment
```sh
conda create -n TWMD-b python=3.7
conda activate TWMD-b
```
#### Install pytorch, transformers, and other packages
```sh
conda install pytorch=1.6 torchvision cudatoolkit=10.1 -c pytorch
conda install numpy scipy
pip install transformers==3.0.2 jsonlines tqdm
```
### Datasets
The STS dataset can also be obtained following the [SBERT-WK](https://github.com/BinWang28/SBERT-WK-Sentence-Embedding) repo, and the WMT 15-18 can also be obtained following the [BLEURT](https://github.com/google-research/bleurt) repository.
### Usage
#### Reproducing our results
The sample codes that recreates our results on STS and WMT datasets using various metrics, with and without centering, can be found in `run_STS.py` and `run_wmt.py`.
We provided metrics `SBERT`, `CKA`, `BERTscore`, `TWMD` and `TRWMD`. Our `MoverScore` implementation follows the [MoverScore](https://github.com/AIPHES/emnlp19-moverscore) repository
For example, running
```sh
python run_wmt.py --model_name roberta-base --use_metric TWMD --use_batch_center True --use_correlation pearsonr --year 17
```
with various flags reproduces our WMT-17 results.
#### Some important parameters and flags for the `Scorer` class
| Flags | Explanation | Default Value |
|:----------:|:------------:|:----------------------------:|
| model-name | model names from the [transformers](https://huggingface.co/models) | bert-base-uncased |
| layer-use | layer embedding used for evaluation | 10 (base models)
19 (large models) |
| ref | reference sentences | None |
| cand | candidate sentences | None |
| gs | ground truth of similarity | None
| metric | use metrics from our metric list | TWMD |
| batch_center | applying batch centering to the word vectors | True |
| batch_size | batch size for evaluation | 32 |
| temp | Temperature for TWMD and TRWMD | 0.1 (TWMD-batch)
0.15 (TRWMD-batch)
0.02 (TWMD/TRWMD)
近期下载者：
相关文件：
收藏者：