Panic: Encoder and decoder must be of equal length; maybe you had duplicate token indices in your encoder?

[chris-monardo]

My team's CI occasionally hits this issue with the GPT OSS encoding.
Is this likely a network download issue? If so integrity checks would be helpful.
Improved debug messages would help too.

thread '<unnamed>' panicked at src/tiktoken.rs:483:9:
Encoder and decoder must be of equal length; maybe you had duplicate token indices in your encoder?

[chris-monardo]

I see there is an integrity check. Maybe related to the race condition in CI issue if tests download the file in parallel. #35