godot-gpt-tokenizer
所属分类：自动编程
开发工具：GDScript
文件大小：0KB
下载次数：0
上传日期：2023-05-03 00:57:08
上 传 者：
sh-1993
说明： OpenAI.的GPT和Codex模型的简单Godot标记器。，
(Simple Godot Tokenizer for GPT and Codex Models by OpenAI.,)
文件列表:
tokenizer/
LICENSE
example.gd
example.tscn
icon.svg
icon.svg.import
project.godot
# Godot GPT Tokenizer
Simple Godot Tokenizer for GPT and Codex Models by OpenAI.
It is a simple conversion of the [GPT-3-Encoder](https://github.com/latitudegames/GPT-3-Encoder) to Godot4.
## Example:
```
extends TextEdit
var tokenizer = Tokenizer.new()
func _ready() -> void:
text_changed.connect(_on_text_changed)
func _on_text_changed():
if len(text) == 0:
return
var encoded = tokenizer.encode(text) # encode text
var token_count = tokenizer.token_count(text) # calc token size
var decoded = tokenizer.decode(encoded) # decode encoded array
# calc token counts from mutliple messages
var messages = [{"assist": "Hello World! 1"}, {"user": "Hello World! 2"}]
var messages_token_count = tokenizer.token_count_from_messages(messages)
print("Encode: " + str(encoded))
print("Token Count: " + str(token_count))
print("Decoded: " + decoded)
print("Token Count from messages: " + str(messages_token_count))
```
近期下载者：
相关文件：
收藏者：