How could you add parallelism to make the encoding faster?

[smahm006]

On line 140-141 of lib.rs, there is a comment where the author mentions he tried threading with rayon but noticed it wasn't much faster than python threads.

Currently the python version gets me the token length in ~0.26 seconds while this crate takes ~1.8 seconds so I propose we should add back threading to speed up the process.

Now I am still a bit new to Rust so this post is more to bring suggestions on how would we go integrating threading?

[zurawiki]

Good points. How are you benchmarking these times today?

There is some overhead with the pyo3 code as it stands and I hope we can optimize that away once openai/tiktoken#40 and openai/tiktoken#50 land

[oslfmt]

I'd like to contribute to this issue. Is this project still active?

[zurawiki]

Issue is open and the project is active! :) Happy to advise / review any PRs

[oslfmt]

Sweet! Besides this issue, any other notable issues/enhancements to work on? Gonna take a closer look tomorrow.

[zurawiki]

Great to hear! Issues that can be worked on are listed here in the GitHub issues. I'd recommend tackling each issue one at a time. You can comment on each issue that you're interested in working on.

[ellipsis-dev]

Sorry, BitBuilder couldn't generate a pull request for you. Please try again later. (wflow_xDf68BFfsE8dDc4G) 🤖