endergebnisandCopilot
4b55a41ece
- Global `--exclude <glob>` (repeatable) on search symbols/text and find dead/dead-symbols, with built-in defaults (vendor/, node_modules/, *.min.*, dist/, build/). Minified/vendored assets stay indexed for import edges but are filtered from query results (bootstrap.min.css noise: 90% of dead symbols -> gone). - `detect` CLI / `codebase_detect` MCP tool: probe a directory's languages and stack from file extensions and manifest files before indexing. - `inspect bundle` CLI / `inspect_bundle` MCP tool: symbol source + direct callers + direct importers in one call (3 round-trips -> 1). - find dead-symbols: skip CSS files and structural kinds (package, heading*) as non-callable; extend framework-wired detection with JUnit/Spring annotations (@Test, @Service, @Component, ...) and pytest/JUnit test-class naming (Test*, test_*, *Test, *Tests), plus `test_` methods reported as `member` kind. - README: correct CSS statement, document new tools/limits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
222 lines
9.1 KiB
Markdown
222 lines
9.1 KiB
Markdown
# rustytokens
|
||
|
||
A native Rust CLI + MCP server that indexes a codebase with
|
||
[`universal-ctags`](https://github.com/universal-ctags/ctags) and lets an AI
|
||
coding agent retrieve **symbols and functions** instead of whole files — saving
|
||
context tokens.
|
||
|
||
Language coverage (via ctags): Rust, Java, Python, C/C++, TypeScript,
|
||
JavaScript, HTML, C#. HTML import edges are extracted directly (not via ctags),
|
||
and `<link>`/`<script>`/`<img>` references to local `.css`/`.js`/image files are
|
||
tracked as imports. universal-ctags (5.9.0+) *does* index CSS selectors as
|
||
symbols, so `search symbols` can return them — `find dead-symbols` skips CSS
|
||
files and structural kinds (selectors/`package`/`heading*` are not callable
|
||
symbols) and the default `--exclude` set filters minified/generated assets.
|
||
|
||
`rustytokens` is an independent, clean-room implementation. It does not share
|
||
code with any AGPL-licensed project.
|
||
|
||
## Prerequisites
|
||
|
||
- Rust (MSRV: current stable; built and tested on nightly)
|
||
- `universal-ctags` on `PATH` (`ctags --version` must report Universal Ctags)
|
||
|
||
```bash
|
||
# Debian/Ubuntu
|
||
sudo apt install universal-ctags
|
||
# macOS
|
||
brew install universal-ctags
|
||
```
|
||
|
||
No system SQLite needed — `rusqlite` bundles SQLite with FTS5.
|
||
|
||
## Build
|
||
|
||
```bash
|
||
cargo build --release
|
||
# binary at target/release/rustytokens
|
||
```
|
||
|
||
## Commands
|
||
|
||
| Command | Purpose |
|
||
|---|---|
|
||
| `index create <path>` | Index a project from scratch |
|
||
| `index update <path>` | Re-index only changed files (content-hash diff) |
|
||
| `index file <file>` | Re-index a single file |
|
||
| `detect [path]` | Detect languages/stack at a path before indexing (no index needed) |
|
||
| `search symbols <query> [--include-source] [--limit n] [--exclude glob]` | Symbol search (exact > prefix > full-text > substring) |
|
||
| `search text <query> [--limit n] [--exclude glob]` | Full-text search with line numbers, read live from disk |
|
||
| `inspect symbol <id\|name>` | Source of a single symbol (id or exact name) |
|
||
| `inspect bundle <id\|name>` | Symbol source + direct callers + direct importers in one call |
|
||
| `inspect outline <file>` | Symbol outline of a file |
|
||
| `inspect file <file> [--lines a-b]` | Read a file or line range |
|
||
| `inspect tree [--depth n]` | Indexed file tree |
|
||
| `inspect blast-radius <file> [--depth n]` | Files transitively affected by a change |
|
||
| `find importers <file>` | Who imports this file |
|
||
| `find dead [--exclude glob]` | Files nobody imports (entry points excluded) |
|
||
| `find dead-symbols [--exclude glob]` | Symbols never referenced (framework-wired entry points excluded) |
|
||
| `find callers <id\|name>` | Symbols that call the given symbol |
|
||
| `find references <identifier>` | Code lines referencing an identifier (definitions/strings/comments excluded) |
|
||
| `inspect cycles` | Circular imports |
|
||
| `serve` | MCP server over stdio |
|
||
| `stats` | Index statistics |
|
||
|
||
`--exclude` is a repeatable glob; it adds to built-in defaults (`vendor/`,
|
||
`node_modules/`, `*.min.*`, `dist/`, `build/`).
|
||
|
||
Global options: `--project <dir>` (default `.`), `--format json|table`
|
||
(default `json`), `--compact`.
|
||
|
||
## Examples
|
||
|
||
```bash
|
||
# Index a project
|
||
rustytokens index create ./myproject
|
||
|
||
# Find the `authenticate` function
|
||
rustytokens --project ./myproject search symbols authenticate
|
||
|
||
# Get only that function's source (id from the search result)
|
||
rustytokens --project ./myproject inspect symbol 42
|
||
|
||
# Which files import auth.rs?
|
||
rustytokens --project ./myproject find importers src/auth.rs
|
||
|
||
# What's dead / circular?
|
||
rustytokens --project ./myproject find dead
|
||
rustytokens --project ./myproject inspect cycles
|
||
|
||
# Who calls `authenticate` (id from `search symbols`) and where is it referenced?
|
||
rustytokens --project ./myproject find callers 42
|
||
rustytokens --project ./myproject find references authenticate
|
||
|
||
# Human-readable output
|
||
rustytokens --project ./myproject --format table search symbols authenticate
|
||
```
|
||
|
||
## MCP integration
|
||
|
||
Register the binary as a stdio MCP server (e.g. in Claude Code / Finir
|
||
`mcp.json`):
|
||
|
||
```json
|
||
{
|
||
"mcpServers": {
|
||
"rustytokens": {
|
||
"command": "/path/to/rustytokens",
|
||
"args": ["--project", "/path/to/your/project", "serve"]
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
Exposed tools: `index_create`, `index_update`, `index_file`,
|
||
`search_symbols` (with `include_source` and `limit`), `search_text` (with
|
||
`limit`), `inspect_symbol`, `inspect_outline`, `inspect_file`, `inspect_tree`,
|
||
`find_importers`, `find_dead`, `find_dead_symbols`, `find_callers`,
|
||
`find_references`, `inspect_cycles`, `inspect_blast_radius`, `stats`.
|
||
|
||
## Why it saves tokens
|
||
|
||
Instead of sending a whole file, the agent fetches only the symbol it needs.
|
||
Measured on `rustytokens`' own `src/ops.rs` (396 lines, ~11.7 KB):
|
||
|
||
- Whole file: 396 lines, 11,675 chars (~2,900 tokens)
|
||
- `inspect symbol` for `build_tree`: 14 lines, 437 chars (~110 tokens)
|
||
|
||
That single retrieval is ~96% smaller than reading the file. Across a large
|
||
codebase and a long session, per-symbol retrieval keeps context focused on the
|
||
code actually being edited.
|
||
|
||
## Performance
|
||
|
||
Measured on a Xeon E5-2667 v3 @ 3.2 GHz (16 cores), `cargo build --release`,
|
||
`universal-ctags` on `PATH`. Wall time, CPU time and peak RSS via
|
||
`/usr/bin/time -v`.
|
||
|
||
| Operation | Repo (files) | Wall | CPU | Peak RSS |
|
||
|---|---|---|---|---|
|
||
| `index create` | nlohmann/json (497) | 1.61 s | 0.98 s | 22 MB |
|
||
| `index create` | gson (264) | 0.92 s | 0.46 s | 13 MB |
|
||
| `index create` (warm) | nlohmann/json (497) | 1.55 s | — | — |
|
||
| `index update` (no-op) | nlohmann/json (497) | 0.02 s | — | — |
|
||
| `search symbols` | — | 8.7 ms | — | 7.8 MB |
|
||
| `find dead` | — | 4.4 ms | — | — |
|
||
| `inspect cycles` | — | 3.7 ms | — | — |
|
||
|
||
Notes:
|
||
|
||
- `index update` is ~50× faster than a full rebuild when nothing changed: it
|
||
hashes every file in parallel (rayon) and re-indexes only changed ones, while
|
||
`index create` drops and rebuilds the whole database. A warm `index create`
|
||
(≈1.55 s) shows that hashing dominates the cost.
|
||
- Hardware cycle counters were not available (`perf` tools missing on this
|
||
kernel), so cycle figures are estimates: CPU time × 3.2 GHz nominal.
|
||
|
||
## Scope and known gaps
|
||
|
||
`rustytokens` covers the same *core* retrieval surface for Java/Python/Rust/
|
||
C/C++/HTML as `toktoken`, but intentionally implements a smaller tool set: 19
|
||
tools vs `toktoken`'s 27. The following `toktoken` tools have **no equivalent
|
||
here yet** (deliberately, not an oversight):
|
||
|
||
- `inspect_dependencies`, `inspect_hierarchy` (graph views)
|
||
- `search_similar`, `search_cooccurrence` (similarity search)
|
||
- GitHub indexing and centrality ranking
|
||
|
||
Add them only if the agent workflow actually needs them; they are not required
|
||
for the "roughly cover Java/Python/Rust/C/C++/HTML" goal.
|
||
|
||
Known limits that are by design, not bugs:
|
||
|
||
- universal-ctags (5.9.0+) indexes CSS selectors as symbols, so `search
|
||
symbols` may return them. `find dead-symbols` skips CSS files and structural
|
||
kinds (`package`, `heading*`) — selectors/declarations are not callable
|
||
symbols — and the default `--exclude` set drops minified assets.
|
||
- `find dead` over-counts for C/C++ header-only entry points (documented
|
||
conservatism); treat results as candidates, not verdicts.
|
||
- `find callers` / `find references` are word-boundary heuristics: they match a
|
||
symbol's name in text, so same-name symbols in unrelated files can produce
|
||
false positives (`find callers` narrows this via the import graph).
|
||
- `find dead-symbols` treats pytest `test_*` functions, `Test*`/`test_*` test
|
||
classes, `#[test]`-wired Rust tests, JUnit/Spring annotations (`@Test`,
|
||
`@Service`, `@Component`, …) and other framework-decorated symbols
|
||
(FastAPI/Flask routes, Celery tasks, click/typer commands, …) as live entry
|
||
points. A pure name-scan cannot see these call sites, so this is a documented
|
||
heuristic.
|
||
- `find references` filters the definition line and occurrences inside strings
|
||
and comments; multi-line docstrings/block comments are approximated
|
||
line-by-line, so a symbol name inside a long string block may still appear.
|
||
|
||
## Development
|
||
|
||
```bash
|
||
cargo fmt
|
||
cargo clippy
|
||
cargo test --release # unit (in-memory SQLite) + CLI integration tests
|
||
```
|
||
|
||
Test fixtures live in `tests/fixtures/` (own small Rust/Python/TS repos).
|
||
|
||
## Licensing
|
||
|
||
`rustytokens` is an independent, clean-room implementation — no code is taken
|
||
from any AGPL-licensed project. The project's own source is licensed under the
|
||
MIT License (see `LICENSE`); `license = "MIT"` is set in `Cargo.toml`.
|
||
|
||
Dependencies are all permissive (MIT / Apache-2.0 / BSL-1.0 / Unlicense):
|
||
`clap`, `rusqlite`, `ignore`, `rayon`, `xxhash-rust`, `serde`, `serde_json`,
|
||
`anyhow`, `regex`. None is copyleft — commercial use and closed-source
|
||
distribution are fine. The only obligation is to ship each crate's copyright
|
||
notice and license text (e.g. a `THIRD_PARTY_NOTICES.md`).
|
||
|
||
`universal-ctags` is GPL-licensed, but `rustytokens` invokes it only as an
|
||
external program over `PATH` (subprocess + stdio, no linking, no bundling). That
|
||
keeps it a separate program and does not impose GPL on `rustytokens` itself. Do
|
||
not bundle the `ctags` binary into a distributed product — that is what would
|
||
trigger GPL obligations for that binary. Keep it a user-installed prerequisite.
|
||
|
||
Naming: `rustytokens` is distinct from `toktoken`. If this goes commercial,
|
||
avoid "toktoken-compatible" or similar wording in marketing.
|