| Index | index by Group | index by Distribution | index by Vendor | index by creation date | index by Name | Mirrors | Help | Search |
| Name: python-tokenizers | Distribution: OpenMandriva Lx |
| Version: 0.23.1 | Vendor: OpenMandriva |
| Release: 1 | Build date: Mon Aug 17 22:51:02 2026 |
| Group: Development/Python | Build host: altra-6.openmandriva.org |
| Size: 0 | Source RPM: python-tokenizers-0.23.1-1.src.rpm |
| Packager: bero <bero@lindev.ch> | |
| Url: https://github.com/huggingface/tokenizers | |
| Summary: Fast state-of-the-art tokenizers for research and production | |
Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Python bindings over the Rust implementation.
Apache-2.0
/usr/lib64/python3.14/site-packages/tokenizers /usr/lib64/python3.14/site-packages/tokenizers-0.23.1.dist-info /usr/lib64/python3.14/site-packages/tokenizers-0.23.1.dist-info/INSTALLER /usr/lib64/python3.14/site-packages/tokenizers-0.23.1.dist-info/METADATA /usr/lib64/python3.14/site-packages/tokenizers-0.23.1.dist-info/RECORD /usr/lib64/python3.14/site-packages/tokenizers-0.23.1.dist-info/REQUESTED /usr/lib64/python3.14/site-packages/tokenizers-0.23.1.dist-info/WHEEL /usr/lib64/python3.14/site-packages/tokenizers-0.23.1.dist-info/direct_url.json /usr/lib64/python3.14/site-packages/tokenizers/__init__.py /usr/lib64/python3.14/site-packages/tokenizers/__init__.pyi /usr/lib64/python3.14/site-packages/tokenizers/__pycache__ /usr/lib64/python3.14/site-packages/tokenizers/__pycache__/__init__.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/decoders /usr/lib64/python3.14/site-packages/tokenizers/decoders.pyi /usr/lib64/python3.14/site-packages/tokenizers/decoders/__init__.py /usr/lib64/python3.14/site-packages/tokenizers/decoders/__pycache__ /usr/lib64/python3.14/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/implementations /usr/lib64/python3.14/site-packages/tokenizers/implementations/__init__.py /usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__ /usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/implementations/base_tokenizer.py /usr/lib64/python3.14/site-packages/tokenizers/implementations/bert_wordpiece.py /usr/lib64/python3.14/site-packages/tokenizers/implementations/byte_level_bpe.py /usr/lib64/python3.14/site-packages/tokenizers/implementations/char_level_bpe.py /usr/lib64/python3.14/site-packages/tokenizers/implementations/sentencepiece_bpe.py /usr/lib64/python3.14/site-packages/tokenizers/implementations/sentencepiece_unigram.py /usr/lib64/python3.14/site-packages/tokenizers/models /usr/lib64/python3.14/site-packages/tokenizers/models.pyi /usr/lib64/python3.14/site-packages/tokenizers/models/__init__.py /usr/lib64/python3.14/site-packages/tokenizers/models/__pycache__ /usr/lib64/python3.14/site-packages/tokenizers/models/__pycache__/__init__.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/normalizers /usr/lib64/python3.14/site-packages/tokenizers/normalizers.pyi /usr/lib64/python3.14/site-packages/tokenizers/normalizers/__init__.py /usr/lib64/python3.14/site-packages/tokenizers/normalizers/__pycache__ /usr/lib64/python3.14/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers /usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers.pyi /usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__init__.py /usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__ /usr/lib64/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/processors /usr/lib64/python3.14/site-packages/tokenizers/processors.pyi /usr/lib64/python3.14/site-packages/tokenizers/processors/__init__.py /usr/lib64/python3.14/site-packages/tokenizers/processors/__pycache__ /usr/lib64/python3.14/site-packages/tokenizers/processors/__pycache__/__init__.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/py.typed /usr/lib64/python3.14/site-packages/tokenizers/tokenizers.abi3.so /usr/lib64/python3.14/site-packages/tokenizers/tokenizers.pyi /usr/lib64/python3.14/site-packages/tokenizers/tools /usr/lib64/python3.14/site-packages/tokenizers/tools/__init__.py /usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__ /usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__/__init__.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-314.pyc /usr/lib64/python3.14/site-packages/tokenizers/tools/visualizer-styles.css /usr/lib64/python3.14/site-packages/tokenizers/tools/visualizer.py /usr/lib64/python3.14/site-packages/tokenizers/trainers /usr/lib64/python3.14/site-packages/tokenizers/trainers.pyi /usr/lib64/python3.14/site-packages/tokenizers/trainers/__init__.py /usr/lib64/python3.14/site-packages/tokenizers/trainers/__init__.pyi /usr/lib64/python3.14/site-packages/tokenizers/trainers/__pycache__ /usr/lib64/python3.14/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-314.pyc /usr/share/doc/python-tokenizers /usr/share/doc/python-tokenizers/README.md /usr/share/licenses/python-tokenizers /usr/share/licenses/python-tokenizers/LICENSE /usr/share/licenses/python-tokenizers/LICENSES.dependencies
Generated by rpm2html 1.8.1
Fabrice Bellet, Tue Aug 18 22:11:28 2026