Add --exclude, detect/inspect_bundle tools, Spring/test entry-point detection
- Global `--exclude <glob>` (repeatable) on search symbols/text and find dead/dead-symbols, with built-in defaults (vendor/, node_modules/, *.min.*, dist/, build/). Minified/vendored assets stay indexed for import edges but are filtered from query results (bootstrap.min.css noise: 90% of dead symbols -> gone). - `detect` CLI / `codebase_detect` MCP tool: probe a directory's languages and stack from file extensions and manifest files before indexing. - `inspect bundle` CLI / `inspect_bundle` MCP tool: symbol source + direct callers + direct importers in one call (3 round-trips -> 1). - find dead-symbols: skip CSS files and structural kinds (package, heading*) as non-callable; extend framework-wired detection with JUnit/Spring annotations (@Test, @Service, @Component, ...) and pytest/JUnit test-class naming (Test*, test_*, *Test, *Tests), plus `test_` methods reported as `member` kind. - README: correct CSS statement, document new tools/limits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
1 parent
b5389341a1
commit
4b55a41ece
9 files changed
+452
-41
No files matched your search
@@ -8,8 +8,10 @@ context tokens.
|
||||
Language coverage (via ctags): Rust, Java, Python, C/C++, TypeScript,
|
||||
JavaScript, HTML, C#. HTML import edges are extracted directly (not via ctags),
|
||||
and `<link>`/`<script>`/`<img>` references to local `.css`/`.js`/image files are
|
||||
tracked as imports. CSS files are discovered and import-tracked, but ctags has
|
||||
no CSS parser — `search symbols` returns nothing for `.css` files.
|
||||
tracked as imports. universal-ctags (5.9.0+) *does* index CSS selectors as
|
||||
symbols, so `search symbols` can return them — `find dead-symbols` skips CSS
|
||||
files and structural kinds (selectors/`package`/`heading*` are not callable
|
||||
symbols) and the default `--exclude` set filters minified/generated assets.
|
||||
|
||||
`rustytokens` is an independent, clean-room implementation. It does not share
|
||||
code with any AGPL-licensed project.
|
||||
@@ -42,22 +44,27 @@ cargo build --release
|
||||
| `index create <path>` | Index a project from scratch |
|
||||
| `index update <path>` | Re-index only changed files (content-hash diff) |
|
||||
| `index file <file>` | Re-index a single file |
|
||||
| `search symbols <query> [--include-source] [--limit n]` | Symbol search (exact > prefix > full-text > substring) |
|
||||
| `search text <query> [--limit n]` | Full-text search with line numbers, read live from disk |
|
||||
| `detect [path]` | Detect languages/stack at a path before indexing (no index needed) |
|
||||
| `search symbols <query> [--include-source] [--limit n] [--exclude glob]` | Symbol search (exact > prefix > full-text > substring) |
|
||||
| `search text <query> [--limit n] [--exclude glob]` | Full-text search with line numbers, read live from disk |
|
||||
| `inspect symbol <id\|name>` | Source of a single symbol (id or exact name) |
|
||||
| `inspect bundle <id\|name>` | Symbol source + direct callers + direct importers in one call |
|
||||
| `inspect outline <file>` | Symbol outline of a file |
|
||||
| `inspect file <file> [--lines a-b]` | Read a file or line range |
|
||||
| `inspect tree [--depth n]` | Indexed file tree |
|
||||
| `inspect blast-radius <file> [--depth n]` | Files transitively affected by a change |
|
||||
| `find importers <file>` | Who imports this file |
|
||||
| `find dead` | Files nobody imports (entry points excluded) |
|
||||
| `find dead-symbols` | Symbols never referenced (framework-wired entry points excluded) |
|
||||
| `find dead [--exclude glob]` | Files nobody imports (entry points excluded) |
|
||||
| `find dead-symbols [--exclude glob]` | Symbols never referenced (framework-wired entry points excluded) |
|
||||
| `find callers <id\|name>` | Symbols that call the given symbol |
|
||||
| `find references <identifier>` | Code lines referencing an identifier (definitions/strings/comments excluded) |
|
||||
| `inspect cycles` | Circular imports |
|
||||
| `serve` | MCP server over stdio |
|
||||
| `stats` | Index statistics |
|
||||
|
||||
`--exclude` is a repeatable glob; it adds to built-in defaults (`vendor/`,
|
||||
`node_modules/`, `*.min.*`, `dist/`, `build/`).
|
||||
|
||||
Global options: `--project <dir>` (default `.`), `--format json|table`
|
||||
(default `json`), `--compact`.
|
||||
|
||||
@@ -150,11 +157,11 @@ Notes:
|
||||
## Scope and known gaps
|
||||
|
||||
`rustytokens` covers the same *core* retrieval surface for Java/Python/Rust/
|
||||
C/C++/HTML as `toktoken`, but intentionally implements a smaller tool set: 17
|
||||
C/C++/HTML as `toktoken`, but intentionally implements a smaller tool set: 19
|
||||
tools vs `toktoken`'s 27. The following `toktoken` tools have **no equivalent
|
||||
here yet** (deliberately, not an oversight):
|
||||
|
||||
- `inspect_bundle`, `inspect_dependencies`, `inspect_hierarchy` (graph views)
|
||||
- `inspect_dependencies`, `inspect_hierarchy` (graph views)
|
||||
- `search_similar`, `search_cooccurrence` (similarity search)
|
||||
- GitHub indexing and centrality ranking
|
||||
|
||||
@@ -163,17 +170,21 @@ for the "roughly cover Java/Python/Rust/C/C++/HTML" goal.
|
||||
|
||||
Known limits that are by design, not bugs:
|
||||
|
||||
- `.css` files are discovered and import-tracked, but `ctags` has no CSS
|
||||
parser — `search symbols` returns nothing for `.css`.
|
||||
- universal-ctags (5.9.0+) indexes CSS selectors as symbols, so `search
|
||||
symbols` may return them. `find dead-symbols` skips CSS files and structural
|
||||
kinds (`package`, `heading*`) — selectors/declarations are not callable
|
||||
symbols — and the default `--exclude` set drops minified assets.
|
||||
- `find dead` over-counts for C/C++ header-only entry points (documented
|
||||
conservatism); treat results as candidates, not verdicts.
|
||||
- `find callers` / `find references` are word-boundary heuristics: they match a
|
||||
symbol's name in text, so same-name symbols in unrelated files can produce
|
||||
false positives (`find callers` narrows this via the import graph).
|
||||
- `find dead-symbols` treats pytest `test_*` functions, `#[test]`-wired Rust
|
||||
tests and framework-decorated symbols (FastAPI/Flask routes, Celery tasks,
|
||||
click/typer commands, …) as live entry points. A pure name-scan cannot see
|
||||
these call sites, so this is a documented heuristic.
|
||||
- `find dead-symbols` treats pytest `test_*` functions, `Test*`/`test_*` test
|
||||
classes, `#[test]`-wired Rust tests, JUnit/Spring annotations (`@Test`,
|
||||
`@Service`, `@Component`, …) and other framework-decorated symbols
|
||||
(FastAPI/Flask routes, Celery tasks, click/typer commands, …) as live entry
|
||||
points. A pure name-scan cannot see these call sites, so this is a documented
|
||||
heuristic.
|
||||
- `find references` filters the definition line and occurrences inside strings
|
||||
and comments; multi-line docstrings/block comments are approximated
|
||||
line-by-line, so a symbol name inside a long string block may still appear.
|
||||
|
||||
Reference in new issue
Block a user