tiktoken-go

A Go binding for tiktoken-rs, for tokenizing text with OpenAI models using tiktoken.

Here are some pure Go implementations you might consider using instead of this one:

• https://github.com/pkoukk/tiktoken-go
• https://github.com/tiktoken-go/tokenizer