rustytokens/README.md

# rustytokens

A native Rust CLI + MCP server that indexes a codebase with
[`universal-ctags`](https://github.com/universal-ctags/ctags) and lets an AI
coding agent retrieve **symbols and functions** instead of whole files — saving
context tokens.

Language coverage (via ctags): Rust, Java, Python, C/C++, TypeScript,
JavaScript, HTML. HTML import edges are extracted directly (not via ctags), and
`<link>`/`<script>`/`<img>` references to local `.css`/`.js`/image files are
tracked as imports. CSS files are discovered and import-tracked, but ctags has
no CSS parser — `search symbols` returns nothing for `.css` files.

`rustytokens` is an independent, clean-room implementation. It does not share
code with any AGPL-licensed project.

## Prerequisites

- Rust (MSRV: current stable; built and tested on nightly)
- `universal-ctags` on `PATH` (`ctags --version` must report Universal Ctags)

```bash
# Debian/Ubuntu
sudo apt install universal-ctags
# macOS
brew install universal-ctags
```

No system SQLite needed — `rusqlite` bundles SQLite with FTS5.

## Build

```bash
cargo build --release
# binary at target/release/rustytokens
```

## Commands

| Command | Purpose |
|---|---|
| `index create <path>` | Index a project from scratch |
| `index update <path>` | Re-index only changed files (content-hash diff) |
| `search symbols <query>` | Symbol search by name (exact > prefix > full-text) |
| `search text <query>` | Full-text search over file contents |
| `inspect symbol <id>` | Source of a single symbol |
| `inspect outline <file>` | Symbol outline of a file |
| `inspect file <file> [--lines a-b]` | Read a file or line range |
| `inspect tree [--depth n]` | Indexed file tree |
| `inspect blast-radius <file> [--depth n]` | Files transitively affected by a change |
| `find importers <file>` | Who imports this file |
| `find dead` | Files nobody imports (entry points excluded) |
| `inspect cycles` | Circular imports |
| `serve` | MCP server over stdio |
| `stats` | Index statistics |

Global options: `--project <dir>` (default `.`), `--format json|table`
(default `json`), `--compact`.

## Examples

```bash
# Index a project
rustytokens index create ./myproject

# Find the `authenticate` function
rustytokens --project ./myproject search symbols authenticate

# Get only that function's source (id from the search result)
rustytokens --project ./myproject inspect symbol 42

# Which files import auth.rs?
rustytokens --project ./myproject find importers src/auth.rs

# What's dead / circular?
rustytokens --project ./myproject find dead
rustytokens --project ./myproject inspect cycles

# Human-readable output
rustytokens --project ./myproject --format table search symbols authenticate
```

## MCP integration

Register the binary as a stdio MCP server (e.g. in Claude Code / Finir
`mcp.json`):

```json
{
  "mcpServers": {
    "rustytokens": {
      "command": "/path/to/rustytokens",
      "args": ["--project", "/path/to/your/project", "serve"]
    }
  }
}
```

Exposed tools: `index_create`, `index_update`, `search_symbols`,
`search_text`, `inspect_symbol`, `inspect_outline`, `inspect_file`,
`inspect_tree`, `find_importers`, `find_dead`, `inspect_cycles`,
`inspect_blast_radius`, `stats`.

## Why it saves tokens

Instead of sending a whole file, the agent fetches only the symbol it needs.
Measured on `rustytokens`' own `src/ops.rs` (396 lines, ~11.7 KB):

- Whole file: 396 lines, 11,675 chars (~2,900 tokens)
- `inspect symbol` for `build_tree`: 14 lines, 437 chars (~110 tokens)

That single retrieval is ~96% smaller than reading the file. Across a large
codebase and a long session, per-symbol retrieval keeps context focused on the
code actually being edited.

## Performance

Measured on a Xeon E5-2667 v3 @ 3.2 GHz (16 cores), `cargo build --release`,
`universal-ctags` on `PATH`. Wall time, CPU time and peak RSS via
`/usr/bin/time -v`.

| Operation | Repo (files) | Wall | CPU | Peak RSS |
|---|---|---|---|---|
| `index create` | nlohmann/json (497) | 1.61 s | 0.98 s | 22 MB |
| `index create` | gson (264) | 0.92 s | 0.46 s | 13 MB |
| `index create` (warm) | nlohmann/json (497) | 1.55 s | — | — |
| `index update` (no-op) | nlohmann/json (497) | 0.02 s | — | — |
| `search symbols` | — | 8.7 ms | — | 7.8 MB |
| `find dead` | — | 4.4 ms | — | — |
| `inspect cycles` | — | 3.7 ms | — | — |

Notes:

- `index update` is ~50× faster than a full rebuild when nothing changed: it
  hashes every file in parallel (rayon) and re-indexes only changed ones, while
  `index create` drops and rebuilds the whole database. A warm `index create`
  (≈1.55 s) shows that hashing dominates the cost.
- Hardware cycle counters were not available (`perf` tools missing on this
  kernel), so cycle figures are estimates: CPU time × 3.2 GHz nominal.

## Development

```bash
cargo fmt
cargo clippy
cargo test --release   # unit (in-memory SQLite) + CLI integration tests
```

Test fixtures live in `tests/fixtures/` (own small Rust/Python/TS repos).

## Licensing

`rustytokens` is an independent, clean-room implementation — no code is taken
from any AGPL-licensed project. The project's own source is licensed under the
MIT License (see `LICENSE`); `license = "MIT"` is set in `Cargo.toml`.

Dependencies are all permissive (MIT / Apache-2.0 / BSL-1.0 / Unlicense):
`clap`, `rusqlite`, `ignore`, `rayon`, `xxhash-rust`, `serde`, `serde_json`,
`anyhow`, `regex`. None is copyleft — commercial use and closed-source
distribution are fine. The only obligation is to ship each crate's copyright
notice and license text (e.g. a `THIRD_PARTY_NOTICES.md`).

`universal-ctags` is GPL-licensed, but `rustytokens` invokes it only as an
external program over `PATH` (subprocess + stdio, no linking, no bundling). That
keeps it a separate program and does not impose GPL on `rustytokens` itself. Do
not bundle the `ctags` binary into a distributed product — that is what would
trigger GPL obligations for that binary. Keep it a user-installed prerequisite.

Naming: `rustytokens` is distinct from `toktoken`. If this goes commercial,
avoid "toktoken-compatible" or similar wording in marketing.