rustytokens

A native Rust CLI + MCP server that indexes a codebase with universal-ctags and lets an AI coding agent retrieve symbols and functions instead of whole files — saving context tokens.

Language coverage (via ctags): Rust, Java, Python, C/C++, TypeScript, JavaScript, HTML. HTML import edges are extracted directly (not via ctags), and <link>/<script>/<img> references to local .css/.js/image files are tracked as imports. CSS files are discovered and import-tracked, but ctags has no CSS parser — search symbols returns nothing for .css files.

rustytokens is an independent, clean-room implementation. It does not share code with any AGPL-licensed project.

Prerequisites

• Rust (MSRV: current stable; built and tested on nightly)
• universal-ctags on PATH (ctags --version must report Universal Ctags)

# Debian/Ubuntu
sudo apt install universal-ctags
# macOS
brew install universal-ctags

No system SQLite needed — rusqlite bundles SQLite with FTS5.

Build

cargo build --release
# binary at target/release/rustytokens

Commands

Command	Purpose
index create <path>	Index a project from scratch
index update <path>	Re-index only changed files (content-hash diff)
search symbols <query>	Symbol search by name (exact > prefix > full-text)
search text <query>	Full-text search over file contents
inspect symbol <id>	Source of a single symbol
inspect outline <file>	Symbol outline of a file
inspect file <file> [--lines a-b]	Read a file or line range
inspect tree [--depth n]	Indexed file tree
inspect blast-radius <file> [--depth n]	Files transitively affected by a change
find importers <file>	Who imports this file
find dead	Files nobody imports (entry points excluded)
inspect cycles	Circular imports
serve	MCP server over stdio
stats	Index statistics

Global options: --project <dir> (default .), --format json|table (default json), --compact.

Examples

# Index a project
rustytokens index create ./myproject

# Find the `authenticate` function
rustytokens --project ./myproject search symbols authenticate

# Get only that function's source (id from the search result)
rustytokens --project ./myproject inspect symbol 42

# Which files import auth.rs?
rustytokens --project ./myproject find importers src/auth.rs

# What's dead / circular?
rustytokens --project ./myproject find dead
rustytokens --project ./myproject inspect cycles

# Human-readable output
rustytokens --project ./myproject --format table search symbols authenticate

MCP integration

Register the binary as a stdio MCP server (e.g. in Claude Code / Finir mcp.json):

{
  "mcpServers": {
    "rustytokens": {
      "command": "/path/to/rustytokens",
      "args": ["--project", "/path/to/your/project", "serve"]
    }
  }
}

Exposed tools: index_create, index_update, search_symbols, search_text, inspect_symbol, inspect_outline, inspect_file, inspect_tree, find_importers, find_dead, inspect_cycles, inspect_blast_radius, stats.

Why it saves tokens

Instead of sending a whole file, the agent fetches only the symbol it needs. Measured on rustytokens' own src/ops.rs (396 lines, ~11.7 KB):

• Whole file: 396 lines, 11,675 chars (~2,900 tokens)
• inspect symbol for build_tree: 14 lines, 437 chars (~110 tokens)

That single retrieval is ~96% smaller than reading the file. Across a large codebase and a long session, per-symbol retrieval keeps context focused on the code actually being edited.

Performance

Measured on a Xeon E5-2667 v3 @ 3.2 GHz (16 cores), cargo build --release, universal-ctags on PATH. Wall time, CPU time and peak RSS via /usr/bin/time -v.

Operation	Repo (files)	Wall	CPU	Peak RSS
index create	nlohmann/json (497)	1.61 s	0.98 s	22 MB
index create	gson (264)	0.92 s	0.46 s	13 MB
index create (warm)	nlohmann/json (497)	1.55 s	—	—
index update (no-op)	nlohmann/json (497)	0.02 s	—	—
search symbols	—	8.7 ms	—	7.8 MB
find dead	—	4.4 ms	—	—
inspect cycles	—	3.7 ms	—	—

Notes:

• index update is ~50× faster than a full rebuild when nothing changed: it hashes every file in parallel (rayon) and re-indexes only changed ones, while index create drops and rebuilds the whole database. A warm index create (≈1.55 s) shows that hashing dominates the cost.
• Hardware cycle counters were not available (perf tools missing on this kernel), so cycle figures are estimates: CPU time × 3.2 GHz nominal.

Scope and known gaps

rustytokens covers the same core retrieval surface for Java/Python/Rust/ C/C++/HTML as toktoken, but intentionally implements a smaller tool set: 13 tools vs toktoken's 27. The following toktoken tools have no equivalent here yet (deliberately, not an oversight):

• index_file (reindex a single file)
• find_callers, find_references (cross-reference search)
• inspect_bundle, inspect_dependencies, inspect_hierarchy (graph views)
• search_similar, search_cooccurrence (similarity search)
• GitHub indexing and centrality ranking

Add them only if the agent workflow actually needs them; they are not required for the "roughly cover Java/Python/Rust/C/C++/HTML" goal.

Known limits that are by design, not bugs:

• .css files are discovered and import-tracked, but ctags has no CSS parser — search symbols returns nothing for .css.
• find dead over-counts for C/C++ header-only entry points (documented conservatism); treat results as candidates, not verdicts.

Development

cargo fmt
cargo clippy
cargo test --release   # unit (in-memory SQLite) + CLI integration tests

Test fixtures live in tests/fixtures/ (own small Rust/Python/TS repos).

Licensing

rustytokens is an independent, clean-room implementation — no code is taken from any AGPL-licensed project. The project's own source is licensed under the MIT License (see LICENSE); license = "MIT" is set in Cargo.toml.

Dependencies are all permissive (MIT / Apache-2.0 / BSL-1.0 / Unlicense): clap, rusqlite, ignore, rayon, xxhash-rust, serde, serde_json, anyhow, regex. None is copyleft — commercial use and closed-source distribution are fine. The only obligation is to ship each crate's copyright notice and license text (e.g. a THIRD_PARTY_NOTICES.md).

universal-ctags is GPL-licensed, but rustytokens invokes it only as an external program over PATH (subprocess + stdio, no linking, no bundling). That keeps it a separate program and does not impose GPL on rustytokens itself. Do not bundle the ctags binary into a distributed product — that is what would trigger GPL obligations for that binary. Keep it a user-installed prerequisite.

Naming: rustytokens is distinct from toktoken. If this goes commercial, avoid "toktoken-compatible" or similar wording in marketing.