rustytokens/README.md

# rustytokens

A native Rust CLI + MCP server that indexes a codebase with
[`universal-ctags`](https://github.com/universal-ctags/ctags) and lets an AI
coding agent retrieve **symbols and functions** instead of whole files — saving
context tokens.

Language coverage (via ctags): Rust, Java, Python, C/C++, TypeScript,
JavaScript, HTML, C#. HTML import edges are extracted directly (not via ctags),
and `<link>`/`<script>`/`<img>` references to local `.css`/`.js`/image files are
tracked as imports. CSS files are discovered and import-tracked, but ctags has
no CSS parser — `search symbols` returns nothing for `.css` files.

`rustytokens` is an independent, clean-room implementation. It does not share
code with any AGPL-licensed project.

## Prerequisites

- Rust (MSRV: current stable; built and tested on nightly)
- `universal-ctags` on `PATH` (`ctags --version` must report Universal Ctags)

```bash
# Debian/Ubuntu
sudo apt install universal-ctags
# macOS
brew install universal-ctags
```

No system SQLite needed — `rusqlite` bundles SQLite with FTS5.

## Build

```bash
cargo build --release
# binary at target/release/rustytokens
```

## Commands

| Command | Purpose |
|---|---|
| `index create <path>` | Index a project from scratch |
| `index update <path>` | Re-index only changed files (content-hash diff) |
| `index file <file>` | Re-index a single file |
| `search symbols <query> [--include-source] [--limit n]` | Symbol search (exact > prefix > full-text > substring) |
| `search text <query> [--limit n]` | Full-text search with line numbers, read live from disk |
| `inspect symbol <id\|name>` | Source of a single symbol (id or exact name) |
| `inspect outline <file>` | Symbol outline of a file |
| `inspect file <file> [--lines a-b]` | Read a file or line range |
| `inspect tree [--depth n]` | Indexed file tree |
| `inspect blast-radius <file> [--depth n]` | Files transitively affected by a change |
| `find importers <file>` | Who imports this file |
| `find dead` | Files nobody imports (entry points excluded) |
| `find dead-symbols` | Symbols never referenced (framework-wired entry points excluded) |
| `find callers <id\|name>` | Symbols that call the given symbol |
| `find references <identifier>` | Code lines referencing an identifier (definitions/strings/comments excluded) |
| `inspect cycles` | Circular imports |
| `serve` | MCP server over stdio |
| `stats` | Index statistics |

Global options: `--project <dir>` (default `.`), `--format json|table`
(default `json`), `--compact`.

## Examples

```bash
# Index a project
rustytokens index create ./myproject

# Find the `authenticate` function
rustytokens --project ./myproject search symbols authenticate

# Get only that function's source (id from the search result)
rustytokens --project ./myproject inspect symbol 42

# Which files import auth.rs?
rustytokens --project ./myproject find importers src/auth.rs

# What's dead / circular?
rustytokens --project ./myproject find dead
rustytokens --project ./myproject inspect cycles

# Who calls `authenticate` (id from `search symbols`) and where is it referenced?
rustytokens --project ./myproject find callers 42
rustytokens --project ./myproject find references authenticate

# Human-readable output
rustytokens --project ./myproject --format table search symbols authenticate
```

## MCP integration

Register the binary as a stdio MCP server (e.g. in Claude Code / Finir
`mcp.json`):

```json
{
  "mcpServers": {
    "rustytokens": {
      "command": "/path/to/rustytokens",
      "args": ["--project", "/path/to/your/project", "serve"]
    }
  }
}
```

Exposed tools: `index_create`, `index_update`, `index_file`,
`search_symbols` (with `include_source` and `limit`), `search_text` (with
`limit`), `inspect_symbol`, `inspect_outline`, `inspect_file`, `inspect_tree`,
`find_importers`, `find_dead`, `find_dead_symbols`, `find_callers`,
`find_references`, `inspect_cycles`, `inspect_blast_radius`, `stats`.

## Why it saves tokens

Instead of sending a whole file, the agent fetches only the symbol it needs.
Measured on `rustytokens`' own `src/ops.rs` (396 lines, ~11.7 KB):

- Whole file: 396 lines, 11,675 chars (~2,900 tokens)
- `inspect symbol` for `build_tree`: 14 lines, 437 chars (~110 tokens)

That single retrieval is ~96% smaller than reading the file. Across a large
codebase and a long session, per-symbol retrieval keeps context focused on the
code actually being edited.

## Performance

Measured on a Xeon E5-2667 v3 @ 3.2 GHz (16 cores), `cargo build --release`,
`universal-ctags` on `PATH`. Wall time, CPU time and peak RSS via
`/usr/bin/time -v`.

| Operation | Repo (files) | Wall | CPU | Peak RSS |
|---|---|---|---|---|
| `index create` | nlohmann/json (497) | 1.61 s | 0.98 s | 22 MB |
| `index create` | gson (264) | 0.92 s | 0.46 s | 13 MB |
| `index create` (warm) | nlohmann/json (497) | 1.55 s | — | — |
| `index update` (no-op) | nlohmann/json (497) | 0.02 s | — | — |
| `search symbols` | — | 8.7 ms | — | 7.8 MB |
| `find dead` | — | 4.4 ms | — | — |
| `inspect cycles` | — | 3.7 ms | — | — |

Notes:

- `index update` is ~50× faster than a full rebuild when nothing changed: it
  hashes every file in parallel (rayon) and re-indexes only changed ones, while
  `index create` drops and rebuilds the whole database. A warm `index create`
  (≈1.55 s) shows that hashing dominates the cost.
- Hardware cycle counters were not available (`perf` tools missing on this
  kernel), so cycle figures are estimates: CPU time × 3.2 GHz nominal.

## Scope and known gaps

`rustytokens` covers the same *core* retrieval surface for Java/Python/Rust/
C/C++/HTML as `toktoken`, but intentionally implements a smaller tool set: 17
tools vs `toktoken`'s 27. The following `toktoken` tools have **no equivalent
here yet** (deliberately, not an oversight):

- `inspect_bundle`, `inspect_dependencies`, `inspect_hierarchy` (graph views)
- `search_similar`, `search_cooccurrence` (similarity search)
- GitHub indexing and centrality ranking

Add them only if the agent workflow actually needs them; they are not required
for the "roughly cover Java/Python/Rust/C/C++/HTML" goal.

Known limits that are by design, not bugs:

- `.css` files are discovered and import-tracked, but `ctags` has no CSS
  parser — `search symbols` returns nothing for `.css`.
- `find dead` over-counts for C/C++ header-only entry points (documented
  conservatism); treat results as candidates, not verdicts.
- `find callers` / `find references` are word-boundary heuristics: they match a
  symbol's name in text, so same-name symbols in unrelated files can produce
  false positives (`find callers` narrows this via the import graph).
- `find dead-symbols` treats pytest `test_*` functions, `#[test]`-wired Rust
  tests and framework-decorated symbols (FastAPI/Flask routes, Celery tasks,
  click/typer commands, …) as live entry points. A pure name-scan cannot see
  these call sites, so this is a documented heuristic.
- `find references` filters the definition line and occurrences inside strings
  and comments; multi-line docstrings/block comments are approximated
  line-by-line, so a symbol name inside a long string block may still appear.

## Development

```bash
cargo fmt
cargo clippy
cargo test --release   # unit (in-memory SQLite) + CLI integration tests
```

Test fixtures live in `tests/fixtures/` (own small Rust/Python/TS repos).

## Licensing

`rustytokens` is an independent, clean-room implementation — no code is taken
from any AGPL-licensed project. The project's own source is licensed under the
MIT License (see `LICENSE`); `license = "MIT"` is set in `Cargo.toml`.

Dependencies are all permissive (MIT / Apache-2.0 / BSL-1.0 / Unlicense):
`clap`, `rusqlite`, `ignore`, `rayon`, `xxhash-rust`, `serde`, `serde_json`,
`anyhow`, `regex`. None is copyleft — commercial use and closed-source
distribution are fine. The only obligation is to ship each crate's copyright
notice and license text (e.g. a `THIRD_PARTY_NOTICES.md`).

`universal-ctags` is GPL-licensed, but `rustytokens` invokes it only as an
external program over `PATH` (subprocess + stdio, no linking, no bundling). That
keeps it a separate program and does not impose GPL on `rustytokens` itself. Do
not bundle the `ctags` binary into a distributed product — that is what would
trigger GPL obligations for that binary. Keep it a user-installed prerequisite.

Naming: `rustytokens` is distinct from `toktoken`. If this goes commercial,
avoid "toktoken-compatible" or similar wording in marketing.