rustytokens/README.md

rustytokens

A native Rust CLI + MCP server that indexes a codebase with universal-ctags and lets an AI coding agent retrieve symbols and functions instead of whole files — saving context tokens.

Language coverage (via ctags): Rust, Java, Python, C/C++, TypeScript, JavaScript, HTML, C#. HTML import edges are extracted directly (not via ctags), and <link>/<script>/<img> references to local .css/.js/image files are tracked as imports. universal-ctags (5.9.0+) does index CSS selectors as symbols, so search symbols can return them — find dead-symbols skips CSS files and structural kinds (selectors/package/heading* are not callable symbols) and the default --exclude set filters minified/generated assets.

rustytokens is an independent, clean-room implementation. It does not share code with any AGPL-licensed project.

Prerequisites

• Rust (MSRV: current stable; built and tested on nightly)
• universal-ctags on PATH (ctags --version must report Universal Ctags)

# Debian/Ubuntu
sudo apt install universal-ctags
# macOS
brew install universal-ctags

No system SQLite needed — rusqlite bundles SQLite with FTS5.

Build

cargo build --release
# binary at target/release/rustytokens

Commands

Command	Purpose
index create <path>	Index a project from scratch
index update <path>	Re-index only changed files (content-hash diff)
index file <file>	Re-index a single file
detect [path]	Detect languages/stack at a path before indexing (no index needed)
search symbols <query> [--include-source] [--limit n] [--exclude glob]	Symbol search (exact > prefix > full-text > substring)
search text <query> [--limit n] [--exclude glob]	Full-text search with line numbers, read live from disk
inspect symbol <id|name>	Source of a single symbol (id or exact name)
inspect bundle <id|name>	Symbol source + direct callers + direct importers in one call
inspect outline <file>	Symbol outline of a file
inspect file <file> [--lines a-b]	Read a file or line range
inspect tree [--depth n]	Indexed file tree
inspect blast-radius <file> [--depth n]	Files transitively affected by a change
find importers <file>	Who imports this file
find dead [--exclude glob]	Files nobody imports (entry points excluded)
find dead-symbols [--exclude glob]	Symbols never referenced (framework-wired entry points excluded)
find callers <id|name>	Symbols that call the given symbol
find references <identifier>	Code lines referencing an identifier (definitions/strings/comments excluded)
inspect cycles	Circular imports
serve	MCP server over stdio
stats	Index statistics

--exclude is a repeatable glob; it adds to built-in defaults (vendor/, node_modules/, *.min.*, dist/, build/).

Global options: --project <dir> (default .), --format json|table (default json), --compact.

Examples

# Index a project
rustytokens index create ./myproject

# Find the `authenticate` function
rustytokens --project ./myproject search symbols authenticate

# Get only that function's source (id from the search result)
rustytokens --project ./myproject inspect symbol 42

# Which files import auth.rs?
rustytokens --project ./myproject find importers src/auth.rs

# What's dead / circular?
rustytokens --project ./myproject find dead
rustytokens --project ./myproject inspect cycles

# Who calls `authenticate` (id from `search symbols`) and where is it referenced?
rustytokens --project ./myproject find callers 42
rustytokens --project ./myproject find references authenticate

# Human-readable output
rustytokens --project ./myproject --format table search symbols authenticate

MCP integration

Register the binary as a stdio MCP server (e.g. in Claude Code / Finir mcp.json):

{
  "mcpServers": {
    "rustytokens": {
      "command": "/path/to/rustytokens",
      "args": ["--project", "/path/to/your/project", "serve"]
    }
  }
}

Exposed tools: index_create, index_update, index_file, search_symbols (with include_source and limit), search_text (with limit), inspect_symbol, inspect_outline, inspect_file, inspect_tree, find_importers, find_dead, find_dead_symbols, find_callers, find_references, inspect_cycles, inspect_blast_radius, stats.

Why it saves tokens

Instead of sending a whole file, the agent fetches only the symbol it needs. Measured on rustytokens' own src/ops.rs (396 lines, ~11.7 KB):

• Whole file: 396 lines, 11,675 chars (~2,900 tokens)
• inspect symbol for build_tree: 14 lines, 437 chars (~110 tokens)

That single retrieval is ~96% smaller than reading the file. Across a large codebase and a long session, per-symbol retrieval keeps context focused on the code actually being edited.

Performance

Measured on a Xeon E5-2667 v3 @ 3.2 GHz (16 cores), cargo build --release, universal-ctags on PATH. Wall time, CPU time and peak RSS via /usr/bin/time -v.

Operation	Repo (files)	Wall	CPU	Peak RSS
index create	nlohmann/json (497)	1.61 s	0.98 s	22 MB
index create	gson (264)	0.92 s	0.46 s	13 MB
index create (warm)	nlohmann/json (497)	1.55 s	—	—
index update (no-op)	nlohmann/json (497)	0.02 s	—	—
search symbols	—	8.7 ms	—	7.8 MB
find dead	—	4.4 ms	—	—
inspect cycles	—	3.7 ms	—	—

Notes:

• index update is ~50× faster than a full rebuild when nothing changed: it hashes every file in parallel (rayon) and re-indexes only changed ones, while index create drops and rebuilds the whole database. A warm index create (≈1.55 s) shows that hashing dominates the cost.
• Hardware cycle counters were not available (perf tools missing on this kernel), so cycle figures are estimates: CPU time × 3.2 GHz nominal.

Scope and known gaps

rustytokens covers the same core retrieval surface for Java/Python/Rust/ C/C++/HTML as toktoken, but intentionally implements a smaller tool set: 19 tools vs toktoken's 27. The following toktoken tools have no equivalent here yet (deliberately, not an oversight):

• inspect_dependencies, inspect_hierarchy (graph views)
• search_similar, search_cooccurrence (similarity search)
• GitHub indexing and centrality ranking

Add them only if the agent workflow actually needs them; they are not required for the "roughly cover Java/Python/Rust/C/C++/HTML" goal.

Known limits that are by design, not bugs:

• universal-ctags (5.9.0+) indexes CSS selectors as symbols, so search symbols may return them. find dead-symbols skips CSS files and structural kinds (package, heading*) — selectors/declarations are not callable symbols — and the default --exclude set drops minified assets.
• find dead over-counts for C/C++ header-only entry points (documented conservatism); treat results as candidates, not verdicts.
• find callers / find references are word-boundary heuristics: they match a symbol's name in text, so same-name symbols in unrelated files can produce false positives (find callers narrows this via the import graph).
• find dead-symbols treats pytest test_* functions, Test*/test_* test classes, #[test]-wired Rust tests, JUnit/Spring annotations (@Test, @Service, @Component, …) and other framework-decorated symbols (FastAPI/Flask routes, Celery tasks, click/typer commands, …) as live entry points. A pure name-scan cannot see these call sites, so this is a documented heuristic.
• find references filters the definition line and occurrences inside strings and comments; multi-line docstrings/block comments are approximated line-by-line, so a symbol name inside a long string block may still appear.

Development

cargo fmt
cargo clippy
cargo test --release   # unit (in-memory SQLite) + CLI integration tests

Test fixtures live in tests/fixtures/ (own small Rust/Python/TS repos).

Licensing

rustytokens is an independent, clean-room implementation — no code is taken from any AGPL-licensed project. The project's own source is licensed under the MIT License (see LICENSE); license = "MIT" is set in Cargo.toml.

Dependencies are all permissive (MIT / Apache-2.0 / BSL-1.0 / Unlicense): clap, rusqlite, ignore, rayon, xxhash-rust, serde, serde_json, anyhow, regex. None is copyleft — commercial use and closed-source distribution are fine. The only obligation is to ship each crate's copyright notice and license text (e.g. a THIRD_PARTY_NOTICES.md).

universal-ctags is GPL-licensed, but rustytokens invokes it only as an external program over PATH (subprocess + stdio, no linking, no bundling). That keeps it a separate program and does not impose GPL on rustytokens itself. Do not bundle the ctags binary into a distributed product — that is what would trigger GPL obligations for that binary. Keep it a user-installed prerequisite.

Naming: rustytokens is distinct from toktoken. If this goes commercial, avoid "toktoken-compatible" or similar wording in marketing.