Add --exclude, detect/inspect_bundle tools, Spring/test entry-point detection

- Global `--exclude <glob>` (repeatable) on search symbols/text and
  find dead/dead-symbols, with built-in defaults (vendor/, node_modules/,
  *.min.*, dist/, build/). Minified/vendored assets stay indexed for import
  edges but are filtered from query results (bootstrap.min.css noise: 90% of
  dead symbols -> gone).
- `detect` CLI / `codebase_detect` MCP tool: probe a directory's languages and
  stack from file extensions and manifest files before indexing.
- `inspect bundle` CLI / `inspect_bundle` MCP tool: symbol source + direct
  callers + direct importers in one call (3 round-trips -> 1).
- find dead-symbols: skip CSS files and structural kinds (package, heading*)
  as non-callable; extend framework-wired detection with JUnit/Spring
  annotations (@Test, @Service, @Component, ...) and pytest/JUnit test-class
  naming (Test*, test_*, *Test, *Tests), plus `test_` methods reported as
  `member` kind.
- README: correct CSS statement, document new tools/limits.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
endergebnisandCopilot committed 2026-09-08 22:48:45 +02:00
1 parent b5389341a1
commit 4b55a41ece
9 files changed
+452 -41

No files matched your search

+25 -14
View File
@@ -8,8 +8,10 @@ context tokens.
Language coverage (via ctags): Rust, Java, Python, C/C++, TypeScript,
JavaScript, HTML, C#. HTML import edges are extracted directly (not via ctags),
and `<link>`/`<script>`/`<img>` references to local `.css`/`.js`/image files are
tracked as imports. CSS files are discovered and import-tracked, but ctags has
no CSS parser — `search symbols` returns nothing for `.css` files.
tracked as imports. universal-ctags (5.9.0+) *does* index CSS selectors as
symbols, so `search symbols` can return them — `find dead-symbols` skips CSS
files and structural kinds (selectors/`package`/`heading*` are not callable
symbols) and the default `--exclude` set filters minified/generated assets.
`rustytokens` is an independent, clean-room implementation. It does not share
code with any AGPL-licensed project.
@@ -42,22 +44,27 @@ cargo build --release
| `index create <path>` | Index a project from scratch |
| `index update <path>` | Re-index only changed files (content-hash diff) |
| `index file <file>` | Re-index a single file |
| `search symbols <query> [--include-source] [--limit n]` | Symbol search (exact > prefix > full-text > substring) |
| `search text <query> [--limit n]` | Full-text search with line numbers, read live from disk |
| `detect [path]` | Detect languages/stack at a path before indexing (no index needed) |
| `search symbols <query> [--include-source] [--limit n] [--exclude glob]` | Symbol search (exact > prefix > full-text > substring) |
| `search text <query> [--limit n] [--exclude glob]` | Full-text search with line numbers, read live from disk |
| `inspect symbol <id\|name>` | Source of a single symbol (id or exact name) |
| `inspect bundle <id\|name>` | Symbol source + direct callers + direct importers in one call |
| `inspect outline <file>` | Symbol outline of a file |
| `inspect file <file> [--lines a-b]` | Read a file or line range |
| `inspect tree [--depth n]` | Indexed file tree |
| `inspect blast-radius <file> [--depth n]` | Files transitively affected by a change |
| `find importers <file>` | Who imports this file |
| `find dead` | Files nobody imports (entry points excluded) |
| `find dead-symbols` | Symbols never referenced (framework-wired entry points excluded) |
| `find dead [--exclude glob]` | Files nobody imports (entry points excluded) |
| `find dead-symbols [--exclude glob]` | Symbols never referenced (framework-wired entry points excluded) |
| `find callers <id\|name>` | Symbols that call the given symbol |
| `find references <identifier>` | Code lines referencing an identifier (definitions/strings/comments excluded) |
| `inspect cycles` | Circular imports |
| `serve` | MCP server over stdio |
| `stats` | Index statistics |
`--exclude` is a repeatable glob; it adds to built-in defaults (`vendor/`,
`node_modules/`, `*.min.*`, `dist/`, `build/`).
Global options: `--project <dir>` (default `.`), `--format json|table`
(default `json`), `--compact`.
@@ -150,11 +157,11 @@ Notes:
## Scope and known gaps
`rustytokens` covers the same *core* retrieval surface for Java/Python/Rust/
C/C++/HTML as `toktoken`, but intentionally implements a smaller tool set: 17
C/C++/HTML as `toktoken`, but intentionally implements a smaller tool set: 19
tools vs `toktoken`'s 27. The following `toktoken` tools have **no equivalent
here yet** (deliberately, not an oversight):
- `inspect_bundle`, `inspect_dependencies`, `inspect_hierarchy` (graph views)
- `inspect_dependencies`, `inspect_hierarchy` (graph views)
- `search_similar`, `search_cooccurrence` (similarity search)
- GitHub indexing and centrality ranking
@@ -163,17 +170,21 @@ for the "roughly cover Java/Python/Rust/C/C++/HTML" goal.
Known limits that are by design, not bugs:
- `.css` files are discovered and import-tracked, but `ctags` has no CSS
parser — `search symbols` returns nothing for `.css`.
- universal-ctags (5.9.0+) indexes CSS selectors as symbols, so `search
symbols` may return them. `find dead-symbols` skips CSS files and structural
kinds (`package`, `heading*`) — selectors/declarations are not callable
symbols — and the default `--exclude` set drops minified assets.
- `find dead` over-counts for C/C++ header-only entry points (documented
conservatism); treat results as candidates, not verdicts.
- `find callers` / `find references` are word-boundary heuristics: they match a
symbol's name in text, so same-name symbols in unrelated files can produce
false positives (`find callers` narrows this via the import graph).
- `find dead-symbols` treats pytest `test_*` functions, `#[test]`-wired Rust
tests and framework-decorated symbols (FastAPI/Flask routes, Celery tasks,
click/typer commands, …) as live entry points. A pure name-scan cannot see
these call sites, so this is a documented heuristic.
- `find dead-symbols` treats pytest `test_*` functions, `Test*`/`test_*` test
classes, `#[test]`-wired Rust tests, JUnit/Spring annotations (`@Test`,
`@Service`, `@Component`, …) and other framework-decorated symbols
(FastAPI/Flask routes, Celery tasks, click/typer commands, …) as live entry
points. A pure name-scan cannot see these call sites, so this is a documented
heuristic.
- `find references` filters the definition line and occurrences inside strings
and comments; multi-line docstrings/block comments are approximated
line-by-line, so a symbol name inside a long string block may still appear.