Add caller/reference analysis, C# imports, and de-noised search

Agent-workflow tooling and search quality improvements:

- find_callers / find_references (by id or exact name), with caller
  attribution to the innermost enclosing symbol and import-graph narrowing
- index file (single-file re-index) and workspace glob member resolution
- search symbols: exact > prefix > fulltext > substring fallback, inline
  source and line numbers, --limit on all search paths
- search text: reads live from disk (no stale FTS snapshot), collapses
  duplicate snippets, correct multi-word phrase matching
- find dead-symbols: excludes framework-wired entry points (pytest test_*,
  #[test] tests, FastAPI/Flask routes, Celery tasks, click/typer commands)
- find references: filters the definition line and string/comment
  occurrences (incl. Rust raw strings)
- C# import edges via new extract_csharp (using directives)
- richer MCP tool descriptions; brace-in-string tokenizer fix

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
endergebnisandCopilot committed 2026-09-08 19:49:36 +02:00
1 parent a416a9fe52
commit f1531a64d4
12 files changed
+1317 -214

No files matched your search

+29 -12
View File
@@ -6,8 +6,8 @@ coding agent retrieve **symbols and functions** instead of whole files — savin
context tokens.
Language coverage (via ctags): Rust, Java, Python, C/C++, TypeScript,
JavaScript, HTML. HTML import edges are extracted directly (not via ctags), and
`<link>`/`<script>`/`<img>` references to local `.css`/`.js`/image files are
JavaScript, HTML, C#. HTML import edges are extracted directly (not via ctags),
and `<link>`/`<script>`/`<img>` references to local `.css`/`.js`/image files are
tracked as imports. CSS files are discovered and import-tracked, but ctags has
no CSS parser — `search symbols` returns nothing for `.css` files.
@@ -41,15 +41,19 @@ cargo build --release
|---|---|
| `index create <path>` | Index a project from scratch |
| `index update <path>` | Re-index only changed files (content-hash diff) |
| `search symbols <query>` | Symbol search by name (exact > prefix > full-text) |
| `search text <query>` | Full-text search over file contents |
| `inspect symbol <id>` | Source of a single symbol |
| `index file <file>` | Re-index a single file |
| `search symbols <query> [--include-source] [--limit n]` | Symbol search (exact > prefix > full-text > substring) |
| `search text <query> [--limit n]` | Full-text search with line numbers, read live from disk |
| `inspect symbol <id\|name>` | Source of a single symbol (id or exact name) |
| `inspect outline <file>` | Symbol outline of a file |
| `inspect file <file> [--lines a-b]` | Read a file or line range |
| `inspect tree [--depth n]` | Indexed file tree |
| `inspect blast-radius <file> [--depth n]` | Files transitively affected by a change |
| `find importers <file>` | Who imports this file |
| `find dead` | Files nobody imports (entry points excluded) |
| `find dead-symbols` | Symbols never referenced (framework-wired entry points excluded) |
| `find callers <id\|name>` | Symbols that call the given symbol |
| `find references <identifier>` | Code lines referencing an identifier (definitions/strings/comments excluded) |
| `inspect cycles` | Circular imports |
| `serve` | MCP server over stdio |
| `stats` | Index statistics |
@@ -76,6 +80,10 @@ rustytokens --project ./myproject find importers src/auth.rs
rustytokens --project ./myproject find dead
rustytokens --project ./myproject inspect cycles
# Who calls `authenticate` (id from `search symbols`) and where is it referenced?
rustytokens --project ./myproject find callers 42
rustytokens --project ./myproject find references authenticate
# Human-readable output
rustytokens --project ./myproject --format table search symbols authenticate
```
@@ -96,10 +104,11 @@ Register the binary as a stdio MCP server (e.g. in Claude Code / Finir
}
```
Exposed tools: `index_create`, `index_update`, `search_symbols`,
`search_text`, `inspect_symbol`, `inspect_outline`, `inspect_file`,
`inspect_tree`, `find_importers`, `find_dead`, `inspect_cycles`,
`inspect_blast_radius`, `stats`.
Exposed tools: `index_create`, `index_update`, `index_file`,
`search_symbols` (with `include_source` and `limit`), `search_text` (with
`limit`), `inspect_symbol`, `inspect_outline`, `inspect_file`, `inspect_tree`,
`find_importers`, `find_dead`, `find_dead_symbols`, `find_callers`,
`find_references`, `inspect_cycles`, `inspect_blast_radius`, `stats`.
## Why it saves tokens
@@ -141,12 +150,10 @@ Notes:
## Scope and known gaps
`rustytokens` covers the same *core* retrieval surface for Java/Python/Rust/
C/C++/HTML as `toktoken`, but intentionally implements a smaller tool set: 13
C/C++/HTML as `toktoken`, but intentionally implements a smaller tool set: 17
tools vs `toktoken`'s 27. The following `toktoken` tools have **no equivalent
here yet** (deliberately, not an oversight):
- `index_file` (reindex a single file)
- `find_callers`, `find_references` (cross-reference search)
- `inspect_bundle`, `inspect_dependencies`, `inspect_hierarchy` (graph views)
- `search_similar`, `search_cooccurrence` (similarity search)
- GitHub indexing and centrality ranking
@@ -160,6 +167,16 @@ Known limits that are by design, not bugs:
parser — `search symbols` returns nothing for `.css`.
- `find dead` over-counts for C/C++ header-only entry points (documented
conservatism); treat results as candidates, not verdicts.
- `find callers` / `find references` are word-boundary heuristics: they match a
symbol's name in text, so same-name symbols in unrelated files can produce
false positives (`find callers` narrows this via the import graph).
- `find dead-symbols` treats pytest `test_*` functions, `#[test]`-wired Rust
tests and framework-decorated symbols (FastAPI/Flask routes, Celery tasks,
click/typer commands, …) as live entry points. A pure name-scan cannot see
these call sites, so this is a documented heuristic.
- `find references` filters the definition line and occurrences inside strings
and comments; multi-line docstrings/block comments are approximated
line-by-line, so a symbol name inside a long string block may still appear.
## Development