Initial commit: rustytokens — native codebase indexer + MCP server

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

README.md

@@ -0,0 +1,172 @@
+# rustytokens
+
+A native Rust CLI + MCP server that indexes a codebase with
+[`universal-ctags`](https://github.com/universal-ctags/ctags) and lets an AI
+coding agent retrieve **symbols and functions** instead of whole files — saving
+context tokens.
+
+Language coverage (via ctags): Rust, Java, Python, C/C++, TypeScript,
+JavaScript, HTML. HTML import edges are extracted directly (not via ctags), and
+`<link>`/`<script>`/`<img>` references to local `.css`/`.js`/image files are
+tracked as imports. CSS files are discovered and import-tracked, but ctags has
+no CSS parser — `search symbols` returns nothing for `.css` files.
+
+`rustytokens` is an independent, clean-room implementation. It does not share
+code with any AGPL-licensed project.
+
+## Prerequisites
+
+- Rust (MSRV: current stable; built and tested on nightly)
+- `universal-ctags` on `PATH` (`ctags --version` must report Universal Ctags)
+
+```bash
+# Debian/Ubuntu
+sudo apt install universal-ctags
+# macOS
+brew install universal-ctags
+```
+
+No system SQLite needed — `rusqlite` bundles SQLite with FTS5.
+
+## Build
+
+```bash
+cargo build --release
+# binary at target/release/rustytokens
+```
+
+## Commands
+
+| Command | Purpose |
+|---|---|
+| `index create <path>` | Index a project from scratch |
+| `index update <path>` | Re-index only changed files (content-hash diff) |
+| `search symbols <query>` | Symbol search by name (exact > prefix > full-text) |
+| `search text <query>` | Full-text search over file contents |
+| `inspect symbol <id>` | Source of a single symbol |
+| `inspect outline <file>` | Symbol outline of a file |
+| `inspect file <file> [--lines a-b]` | Read a file or line range |
+| `inspect tree [--depth n]` | Indexed file tree |
+| `inspect blast-radius <file> [--depth n]` | Files transitively affected by a change |
+| `find importers <file>` | Who imports this file |
+| `find dead` | Files nobody imports (entry points excluded) |
+| `inspect cycles` | Circular imports |
+| `serve` | MCP server over stdio |
+| `stats` | Index statistics |
+
+Global options: `--project <dir>` (default `.`), `--format json|table`
+(default `json`), `--compact`.
+
+## Examples
+
+```bash
+# Index a project
+rustytokens index create ./myproject
+
+# Find the `authenticate` function
+rustytokens --project ./myproject search symbols authenticate
+
+# Get only that function's source (id from the search result)
+rustytokens --project ./myproject inspect symbol 42
+
+# Which files import auth.rs?
+rustytokens --project ./myproject find importers src/auth.rs
+
+# What's dead / circular?
+rustytokens --project ./myproject find dead
+rustytokens --project ./myproject inspect cycles
+
+# Human-readable output
+rustytokens --project ./myproject --format table search symbols authenticate
+```
+
+## MCP integration
+
+Register the binary as a stdio MCP server (e.g. in Claude Code / Finir
+`mcp.json`):
+
+```json
+{
+  "mcpServers": {
+    "rustytokens": {
+      "command": "/path/to/rustytokens",
+      "args": ["--project", "/path/to/your/project", "serve"]
+    }
+  }
+}
+```
+
+Exposed tools: `index_create`, `index_update`, `search_symbols`,
+`search_text`, `inspect_symbol`, `inspect_outline`, `inspect_file`,
+`inspect_tree`, `find_importers`, `find_dead`, `inspect_cycles`,
+`inspect_blast_radius`, `stats`.
+
+## Why it saves tokens
+
+Instead of sending a whole file, the agent fetches only the symbol it needs.
+Measured on `rustytokens`' own `src/ops.rs` (396 lines, ~11.7 KB):
+
+- Whole file: 396 lines, 11,675 chars (~2,900 tokens)
+- `inspect symbol` for `build_tree`: 14 lines, 437 chars (~110 tokens)
+
+That single retrieval is ~96% smaller than reading the file. Across a large
+codebase and a long session, per-symbol retrieval keeps context focused on the
+code actually being edited.
+
+## Performance
+
+Measured on a Xeon E5-2667 v3 @ 3.2 GHz (16 cores), `cargo build --release`,
+`universal-ctags` on `PATH`. Wall time, CPU time and peak RSS via
+`/usr/bin/time -v`.
+
+| Operation | Repo (files) | Wall | CPU | Peak RSS |
+|---|---|---|---|---|
+| `index create` | nlohmann/json (497) | 1.61 s | 0.98 s | 22 MB |
+| `index create` | gson (264) | 0.92 s | 0.46 s | 13 MB |
+| `index create` (warm) | nlohmann/json (497) | 1.55 s | — | — |
+| `index update` (no-op) | nlohmann/json (497) | 0.02 s | — | — |
+| `search symbols` | — | 8.7 ms | — | 7.8 MB |
+| `find dead` | — | 4.4 ms | — | — |
+| `inspect cycles` | — | 3.7 ms | — | — |
+
+Notes:
+
+- `index update` is ~50× faster than a full rebuild when nothing changed: it
+  hashes every file in parallel (rayon) and re-indexes only changed ones, while
+  `index create` drops and rebuilds the whole database. A warm `index create`
+  (≈1.55 s) shows that hashing dominates the cost.
+- Hardware cycle counters were not available (`perf` tools missing on this
+  kernel), so cycle figures are estimates: CPU time × 3.2 GHz nominal.
+
+## Development
+
+```bash
+cargo fmt
+cargo clippy
+cargo test --release   # unit (in-memory SQLite) + CLI integration tests
+```
+
+Test fixtures live in `tests/fixtures/` (own small Rust/Python/TS repos).
+
+## Licensing
+
+`rustytokens` is an independent, clean-room implementation — no code is taken
+from any AGPL-licensed project, so the project's own source is yours to license
+however you choose (MIT, proprietary, commercial, closed-source). Decide and
+set it once: add a `license` field to `Cargo.toml` and, if you ship the source,
+drop in a `LICENSE` file.
+
+Dependencies are all permissive (MIT / Apache-2.0 / BSL-1.0 / Unlicense):
+`clap`, `rusqlite`, `ignore`, `rayon`, `xxhash-rust`, `serde`, `serde_json`,
+`anyhow`, `regex`. None is copyleft — commercial use and closed-source
+distribution are fine. The only obligation is to ship each crate's copyright
+notice and license text (e.g. a `THIRD_PARTY_NOTICES.md`).
+
+`universal-ctags` is GPL-licensed, but `rustytokens` invokes it only as an
+external program over `PATH` (subprocess + stdio, no linking, no bundling). That
+keeps it a separate program and does not impose GPL on `rustytokens` itself. Do
+not bundle the `ctags` binary into a distributed product — that is what would
+trigger GPL obligations for that binary. Keep it a user-installed prerequisite.
+
+Naming: `rustytokens` is distinct from `toktoken`. If this goes commercial,
+avoid "toktoken-compatible" or similar wording in marketing.