📦 j178 / tiktoken-go-binding

1
2
3
4
5
6
7
8
9# tiktoken-go

A Go binding for [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs), for tokenizing text with OpenAI models using tiktoken.

Here are some pure Go implementations you might consider using instead of this one:

- https://github.com/pkoukk/tiktoken-go
- https://github.com/tiktoken-go/tokenizer