GPT-3-Encoder
所属分类：GPT/ChatGPT
开发工具：JavaScript
文件大小：602KB
下载次数：0
上传日期：2023-04-16 00:44:43
上 传 者：
sh-1993
说明： 用于GPT-2 GPT-3的Javascript BPE编码器解码器
(Javascript BPE Encoder Decoder for GPT-2 / GPT-3
,)
文件列表:
Encoder.js (4000, 2023-01-19)
Encoder.test.js (1161, 2023-01-19)
LICENSE (1066, 2023-01-19)
encoder.json (1042301, 2023-01-19)
encoder.py (3615, 2023-01-19)
example.js (431, 2023-01-19)
index.d.ts (138, 2023-01-19)
index.js (92, 2023-01-19)
jest.config.js (422, 2023-01-19)
package-lock.json (178210, 2023-01-19)
package.json (638, 2023-01-19)
vocab.bpe (456318, 2023-01-19)
# GPT-3-Encoder
Javascript BPE Encoder Decoder for GPT-2 / GPT-3
## About
GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. This is a javascript implementation of OpenAI's original python encoder/decoder which can be found [here](https://github.com/openai/gpt-2)
## Install with npm
```
npm install gpt-3-encoder
```
## Usage
Compatible with Node >= 12
```js
const {encode, decode} = require('gpt-3-encoder')
const str = 'This is an example sentence to try encoding out on!'
const encoded = encode(str)
console.log('Encoded this string looks like: ', encoded)
console.log('We can look at each token and what it represents')
for(let token of encoded){
console.log({token, string: decode([token])})
}
const decoded = decode(encoded)
console.log('We can decode it back into:\n', decoded)
```
近期下载者：
相关文件：
收藏者：