gpt-2-3-tokenizer
所属分类：GPT/ChatGPT
开发工具：JavaScript
文件大小：544KB
下载次数：0
上传日期：2023-03-29 15:45:54
上 传 者：
sh-1993
说明： GPT-2 3令牌化器基于@latitudegames GPT-3编码器，可在浏览器和Deno中工作
(GPT-2/3 tokenizer based on @latitudegames/GPT-3-Encoder that works in the
browser and Deno
,)
文件列表:
encoder.js (1042317, 2023-03-29)
mod.js (3755, 2023-03-29)
vocab.bpe.js (478414, 2023-03-29)
# GPT-2/3 Tokenizer
GPT-2/3 byte pair encoder/decoder/tokenizer based on [@latitudegames/GPT-3-Encoder](https://github.com/latitudegames/GPT-3-Encoder) that works in the browser and Deno.
See also: [JS byte pair encoder for OpenAI's CLIP model](https://github.com/josephrocca/clip-bpe-js).
```js
import {encode, decode} from "https://deno.land/x/gpt_2_3_tokenizer@v0.0.2/mod.js";
let text = "hello world";
console.log(encode(text)); // [258, 187***, 995]
console.log(decode(encode(text))); // "hello world"
```
or:
```js
let mod = await import("https://deno.land/x/gpt_2_3_tokenizer@v0.0.2/mod.js");
mod.encode("hello world"); // [258, 187***, 995]
```
or to include it as a global variable (as if you were importing it with the old script tag style):
```html
```
# License
The [original code is MIT Licensed](https://github.com/latitudegames/GPT-3-Encoder/blob/master/LICENSE) and so are any changes made by this repo.
近期下载者：
相关文件：
收藏者：