Fast • UTF-8 Native • Multi-threaded

Build Tokenizers
Faster than Ever

A modern UTF-8 native tokenizer engine for Python designed for speed, scalability, and simplicity. Built with C++ for performance and exposed through an easy-to-use Python API.

Install Now View GitHub
py_tokenizer Logo Large
Status Optimized
C++ Core Python API

Get Started in Seconds

Integrate blazing fast tokenization into your workflow with zero friction.

Installation

Available on PyPI. Install via pip in your virtual environment.

pip install py_tokenizer_ansh
example.py
from py_tokenizer import engine

engine(
    "/storage/emulated/0/Download/big.txt",  # Input file
    4,                                       # Threads
    True,                                    # Normalize
    False,                                   # Verbose
    100,                                     # Chunk size
    8,                                       # Epochs
    True                                     # Generate vocab
)

This command instantly generates the tokenizer vocabulary (vocabulary.json) and related output files utilizing native multi-threading.

Engineered for Excellence

Every feature is designed to maximize throughput and minimize developer friction.

UTF-8 Native Processing

Correctly handles multilingual Unicode text without breaking UTF-8 characters.

Multi-threaded Processing

Uses multiple CPU threads for faster vocabulary generation and preprocessing.

High Performance

Core engine written in modern C++ for maximum speed and memory efficiency.

Python Friendly

Simple Python interface with minimal setup. Integrates directly into your workflow.

Vocabulary Builder

Automatically creates optimized vocabulary.json from vast text datasets.

Scalable

Engineered to work flawlessly with both small experiments and very large text files.

Clean API

Easy to integrate into Natural Language Processing (NLP) and Artificial Intelligence projects.

Lightweight

Designed to be inherently simple and efficient, without unnecessary bloat or dependencies.

Clean Pipeline Architecture

Raw Text File
UTF-8 Reader
Chunk Processor
Thread Workers
Vocabulary Builder
vocabulary.json

Optimized for Speed

py_tokenizer is built from the ground up to eliminate bottlenecks during text ingestion and vocabulary generation, processing gigabytes of text effortlessly.

  • Modern C++

    Under the hood, memory allocation and string manipulation are handled by highly optimized C++ routines bypassing Python's GIL.

  • Parallel Processing

    Chunk-based reading allows multiple CPU threads to process segments of large files concurrently.

  • Memory Efficient Design

    Streams text efficiently instead of loading massive datasets into RAM entirely, avoiding out-of-memory errors.

  • UTF-8 Native Parsing

    Directly interprets UTF-8 byte streams, ensuring perfect token boundaries across all languages.

API Overview

Choose the interface that fits your project's complexity.

Beginner Friendly

Basic Engine

The simplest interface for quickly processing text files. Suitable for beginners and small-scale projects where advanced tuning is unnecessary.

from py_tokenizer import basic_engine

basic_engine("data.txt")
Recommended

Engine

The advanced processing engine for professional use. Unlocks the full power of the C++ backend.

  • UTF-8 text support
  • Multi-threading
  • Configurable chunks
  • Normalization
  • Adjustable epochs
  • High-speed vocab

Why py_tokenizer?

See how we stack up against standard implementations.

Feature py_tokenizer Typical Tokenizers
UTF-8 Native Sometimes
Multi-threading Limited
Modern C++ Backend Rare
Python API
Vocabulary Generation Limited
Large File Support Depends
Easy Integration

Built for Every Scenario

Versatile enough to handle modern AI development workloads.

Natural Language Processing
Large Dataset Preprocessing
Vocabulary Creation
Language Models
Machine Learning
Research
Chatbots
Educational Projects
AI Experiments

Documentation

Dive deep into the architecture, view advanced API examples, and learn how to customize tokenization for your specific language models. Full documentation is currently being compiled and will be available shortly on ReadTheDocs.

Installation Guide
API Reference
Advanced Tutorials (Coming Soon)

Open Source

py_tokenizer is an open-source project maintained by Ansh Raj. We welcome contributions, feature requests, and bug reports on GitHub.

GitHub Repository

Source code, issues, and contribution guidelines.

View Repository

Developer Portfolio

Explore more tools and projects by Ansh Raj.

Visit Portfolio