Add caller/reference analysis, C# imports, and de-noised search
Agent-workflow tooling and search quality improvements: - find_callers / find_references (by id or exact name), with caller attribution to the innermost enclosing symbol and import-graph narrowing - index file (single-file re-index) and workspace glob member resolution - search symbols: exact > prefix > fulltext > substring fallback, inline source and line numbers, --limit on all search paths - search text: reads live from disk (no stale FTS snapshot), collapses duplicate snippets, correct multi-word phrase matching - find dead-symbols: excludes framework-wired entry points (pytest test_*, #[test] tests, FastAPI/Flask routes, Celery tasks, click/typer commands) - find references: filters the definition line and string/comment occurrences (incl. Rust raw strings) - C# import edges via new extract_csharp (using directives) - richer MCP tool descriptions; brace-in-string tokenizer fix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
endergebnisandCopilot committed
1 parent
a416a9fe52
commit
f1531a64d4
12 files changed
+1313
-210
No files matched your search
@@ -6,8 +6,8 @@ coding agent retrieve **symbols and functions** instead of whole files — savin
|
||||
context tokens.
|
||||
|
||||
Language coverage (via ctags): Rust, Java, Python, C/C++, TypeScript,
|
||||
JavaScript, HTML. HTML import edges are extracted directly (not via ctags), and
|
||||
`<link>`/`<script>`/`<img>` references to local `.css`/`.js`/image files are
|
||||
JavaScript, HTML, C#. HTML import edges are extracted directly (not via ctags),
|
||||
and `<link>`/`<script>`/`<img>` references to local `.css`/`.js`/image files are
|
||||
tracked as imports. CSS files are discovered and import-tracked, but ctags has
|
||||
no CSS parser — `search symbols` returns nothing for `.css` files.
|
||||
|
||||
@@ -41,15 +41,19 @@ cargo build --release
|
||||
|---|---|
|
||||
| `index create <path>` | Index a project from scratch |
|
||||
| `index update <path>` | Re-index only changed files (content-hash diff) |
|
||||
| `search symbols <query>` | Symbol search by name (exact > prefix > full-text) |
|
||||
| `search text <query>` | Full-text search over file contents |
|
||||
| `inspect symbol <id>` | Source of a single symbol |
|
||||
| `index file <file>` | Re-index a single file |
|
||||
| `search symbols <query> [--include-source] [--limit n]` | Symbol search (exact > prefix > full-text > substring) |
|
||||
| `search text <query> [--limit n]` | Full-text search with line numbers, read live from disk |
|
||||
| `inspect symbol <id\|name>` | Source of a single symbol (id or exact name) |
|
||||
| `inspect outline <file>` | Symbol outline of a file |
|
||||
| `inspect file <file> [--lines a-b]` | Read a file or line range |
|
||||
| `inspect tree [--depth n]` | Indexed file tree |
|
||||
| `inspect blast-radius <file> [--depth n]` | Files transitively affected by a change |
|
||||
| `find importers <file>` | Who imports this file |
|
||||
| `find dead` | Files nobody imports (entry points excluded) |
|
||||
| `find dead-symbols` | Symbols never referenced (framework-wired entry points excluded) |
|
||||
| `find callers <id\|name>` | Symbols that call the given symbol |
|
||||
| `find references <identifier>` | Code lines referencing an identifier (definitions/strings/comments excluded) |
|
||||
| `inspect cycles` | Circular imports |
|
||||
| `serve` | MCP server over stdio |
|
||||
| `stats` | Index statistics |
|
||||
@@ -76,6 +80,10 @@ rustytokens --project ./myproject find importers src/auth.rs
|
||||
rustytokens --project ./myproject find dead
|
||||
rustytokens --project ./myproject inspect cycles
|
||||
|
||||
# Who calls `authenticate` (id from `search symbols`) and where is it referenced?
|
||||
rustytokens --project ./myproject find callers 42
|
||||
rustytokens --project ./myproject find references authenticate
|
||||
|
||||
# Human-readable output
|
||||
rustytokens --project ./myproject --format table search symbols authenticate
|
||||
```
|
||||
@@ -96,10 +104,11 @@ Register the binary as a stdio MCP server (e.g. in Claude Code / Finir
|
||||
}
|
||||
```
|
||||
|
||||
Exposed tools: `index_create`, `index_update`, `search_symbols`,
|
||||
`search_text`, `inspect_symbol`, `inspect_outline`, `inspect_file`,
|
||||
`inspect_tree`, `find_importers`, `find_dead`, `inspect_cycles`,
|
||||
`inspect_blast_radius`, `stats`.
|
||||
Exposed tools: `index_create`, `index_update`, `index_file`,
|
||||
`search_symbols` (with `include_source` and `limit`), `search_text` (with
|
||||
`limit`), `inspect_symbol`, `inspect_outline`, `inspect_file`, `inspect_tree`,
|
||||
`find_importers`, `find_dead`, `find_dead_symbols`, `find_callers`,
|
||||
`find_references`, `inspect_cycles`, `inspect_blast_radius`, `stats`.
|
||||
|
||||
## Why it saves tokens
|
||||
|
||||
@@ -141,12 +150,10 @@ Notes:
|
||||
## Scope and known gaps
|
||||
|
||||
`rustytokens` covers the same *core* retrieval surface for Java/Python/Rust/
|
||||
C/C++/HTML as `toktoken`, but intentionally implements a smaller tool set: 13
|
||||
C/C++/HTML as `toktoken`, but intentionally implements a smaller tool set: 17
|
||||
tools vs `toktoken`'s 27. The following `toktoken` tools have **no equivalent
|
||||
here yet** (deliberately, not an oversight):
|
||||
|
||||
- `index_file` (reindex a single file)
|
||||
- `find_callers`, `find_references` (cross-reference search)
|
||||
- `inspect_bundle`, `inspect_dependencies`, `inspect_hierarchy` (graph views)
|
||||
- `search_similar`, `search_cooccurrence` (similarity search)
|
||||
- GitHub indexing and centrality ranking
|
||||
@@ -160,6 +167,16 @@ Known limits that are by design, not bugs:
|
||||
parser — `search symbols` returns nothing for `.css`.
|
||||
- `find dead` over-counts for C/C++ header-only entry points (documented
|
||||
conservatism); treat results as candidates, not verdicts.
|
||||
- `find callers` / `find references` are word-boundary heuristics: they match a
|
||||
symbol's name in text, so same-name symbols in unrelated files can produce
|
||||
false positives (`find callers` narrows this via the import graph).
|
||||
- `find dead-symbols` treats pytest `test_*` functions, `#[test]`-wired Rust
|
||||
tests and framework-decorated symbols (FastAPI/Flask routes, Celery tasks,
|
||||
click/typer commands, …) as live entry points. A pure name-scan cannot see
|
||||
these call sites, so this is a documented heuristic.
|
||||
- `find references` filters the definition line and occurrences inside strings
|
||||
and comments; multi-line docstrings/block comments are approximated
|
||||
line-by-line, so a symbol name inside a long string block may still appear.
|
||||
|
||||
## Development
|
||||
|
||||
|
||||
+26
-5
@@ -44,7 +44,7 @@ pub enum Command {
|
||||
#[command(subcommand)]
|
||||
sub: InspectCommand,
|
||||
},
|
||||
/// Find importers or dead code
|
||||
/// Find importers, callers, references, or dead code
|
||||
Find {
|
||||
#[command(subcommand)]
|
||||
sub: FindCommand,
|
||||
@@ -61,20 +61,35 @@ pub enum IndexCommand {
|
||||
Create { path: PathBuf },
|
||||
/// Re-index only changed files
|
||||
Update { path: PathBuf },
|
||||
/// Re-index a single file (relative or absolute path)
|
||||
File { file: String },
|
||||
}
|
||||
|
||||
#[derive(Subcommand)]
|
||||
pub enum SearchCommand {
|
||||
/// Search symbols by name
|
||||
Symbols { query: String },
|
||||
Symbols {
|
||||
query: String,
|
||||
/// Include the symbol's source inline (skips the inspect_symbol round-trip)
|
||||
#[arg(long)]
|
||||
include_source: bool,
|
||||
/// Maximum number of results (default 200)
|
||||
#[arg(long, default_value_t = 200)]
|
||||
limit: usize,
|
||||
},
|
||||
/// Full-text search over file contents
|
||||
Text { query: String },
|
||||
Text {
|
||||
query: String,
|
||||
/// Maximum number of matching files (default 100)
|
||||
#[arg(long, default_value_t = 100)]
|
||||
limit: usize,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Subcommand)]
|
||||
pub enum InspectCommand {
|
||||
/// Source code of a single symbol by id
|
||||
Symbol { id: i64 },
|
||||
/// Source code of a symbol by id or exact name
|
||||
Symbol { target: String },
|
||||
/// Symbol outline of a file
|
||||
Outline { file: String },
|
||||
/// Read a file or a line range (e.g. --lines 10-40)
|
||||
@@ -104,4 +119,10 @@ pub enum FindCommand {
|
||||
Importers { file: String },
|
||||
/// Files with no importers
|
||||
Dead,
|
||||
/// Symbols never referenced by name outside their definition
|
||||
DeadSymbols,
|
||||
/// Symbols that call the given symbol (by id or exact name)
|
||||
Callers { target: String },
|
||||
/// Lines where an identifier is referenced
|
||||
References { identifier: String },
|
||||
}
|
||||
+8
-6
@@ -1,7 +1,5 @@
|
||||
//! Project configuration: paths, cache dir, ignore patterns, language mapping.
|
||||
|
||||
use std::collections::hash_map::DefaultHasher;
|
||||
use std::hash::{Hash, Hasher};
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
/// Directory names skipped during discovery (own list).
|
||||
@@ -52,6 +50,7 @@ const LANGUAGE_BY_EXT: &[(&str, &str)] = &[
|
||||
("hh", "cpp"),
|
||||
("hxx", "cpp"),
|
||||
("java", "java"),
|
||||
("cs", "csharp"),
|
||||
("html", "html"),
|
||||
("htm", "html"),
|
||||
("css", "css"),
|
||||
@@ -92,10 +91,13 @@ pub fn cache_dir() -> PathBuf {
|
||||
}
|
||||
|
||||
/// Deterministic short key for a project path (used to name the cache DB).
|
||||
///
|
||||
/// xxh3 rather than `DefaultHasher` (SipHash): `DefaultHasher` is explicitly
|
||||
/// not guaranteed stable across Rust releases, which would silently orphan the
|
||||
/// old index on every toolchain upgrade. xxh3 is a fixed, version-independent
|
||||
/// algorithm and is already a dependency.
|
||||
fn project_key(path: &str) -> String {
|
||||
let mut h = DefaultHasher::new();
|
||||
path.hash(&mut h);
|
||||
format!("{:016x}", h.finish())
|
||||
format!("{:016x}", xxhash_rust::xxh3::xxh3_64(path.as_bytes()))
|
||||
}
|
||||
|
||||
/// Path of the SQLite index DB for a project root.
|
||||
@@ -104,7 +106,7 @@ pub fn index_db_path(project: &Path) -> PathBuf {
|
||||
.canonicalize()
|
||||
.unwrap_or_else(|_| project.to_path_buf());
|
||||
let key = project_key(&canon.to_string_lossy());
|
||||
let name = project
|
||||
let name = canon
|
||||
.file_name()
|
||||
.map(|n| n.to_string_lossy().into_owned())
|
||||
.unwrap_or_else(|| "project".to_string());
|
||||
|
||||
@@ -12,12 +12,3 @@ pub fn hash_file(path: &Path) -> Result<String> {
|
||||
f.read_to_end(&mut buf)?;
|
||||
Ok(format!("{:016x}", xxhash_rust::xxh3::xxh3_64(&buf)))
|
||||
}
|
||||
|
||||
pub fn mtime_secs(path: &Path) -> i64 {
|
||||
std::fs::metadata(path)
|
||||
.and_then(|m| m.modified())
|
||||
.ok()
|
||||
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
|
||||
.map(|d| d.as_secs() as i64)
|
||||
.unwrap_or(0)
|
||||
}
|
||||
+62
-12
@@ -26,6 +26,7 @@ pub fn extract(f: &SourceFile) -> Vec<Import> {
|
||||
"c" | "cpp" => extract_c(&content),
|
||||
"java" => extract_java(&content),
|
||||
"html" => extract_html(&content),
|
||||
"csharp" => extract_csharp(&content),
|
||||
_ => Vec::new(),
|
||||
}
|
||||
}
|
||||
@@ -65,11 +66,30 @@ fn extract_python(content: &str) -> Vec<Import> {
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// `from <module> import a, b` — record the module edge (for
|
||||
// symbols re-exported from `<module>/__init__.py`) plus a
|
||||
// best-effort edge per imported name as a potential submodule
|
||||
// (`from finir.lib import auto_code` → `finir.lib.auto_code`).
|
||||
out.push(Import {
|
||||
target: name.to_string(),
|
||||
import_type: "from".into(),
|
||||
levels,
|
||||
});
|
||||
if let Some(after) = rest.split_once("import ") {
|
||||
for n in after.1.split(',') {
|
||||
let n = n.trim().trim_end_matches('\\').trim();
|
||||
// Drop `as alias`; keep only the imported name.
|
||||
let n = n.split_whitespace().next().unwrap_or(n);
|
||||
if n.is_empty() || n == "*" {
|
||||
continue;
|
||||
}
|
||||
out.push(Import {
|
||||
target: format!("{name}.{n}"),
|
||||
import_type: "from".into(),
|
||||
levels,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -103,7 +123,12 @@ fn extract_rust(content: &str) -> Vec<Import> {
|
||||
.trim_start_matches("crate::")
|
||||
.trim_start_matches("self::")
|
||||
.trim_start_matches("super::")
|
||||
.trim_start_matches("::");
|
||||
.trim_start_matches("::")
|
||||
// `use foo as bar;` — keep only the crate/module path.
|
||||
.split(" as ")
|
||||
.next()
|
||||
.unwrap_or(rest)
|
||||
.trim();
|
||||
let mut segs: Vec<&str> = path.split("::").filter(|s| !s.is_empty()).collect();
|
||||
if segs.is_empty() || matches!(segs[0], "std" | "core" | "alloc") {
|
||||
continue;
|
||||
@@ -228,6 +253,31 @@ fn extract_c(content: &str) -> Vec<Import> {
|
||||
out
|
||||
}
|
||||
|
||||
/// C# `using` directives. Namespaces are dotted and map back to paths through
|
||||
/// the resolver's dotted→slashed fallback; `using static X;` and
|
||||
/// `using Alias = X;` reduce to the namespace/type path `X`.
|
||||
fn extract_csharp(content: &str) -> Vec<Import> {
|
||||
let mut out = Vec::new();
|
||||
for line in content.lines() {
|
||||
let l = line.trim();
|
||||
let Some(rest) = l.strip_prefix("using ") else {
|
||||
continue;
|
||||
};
|
||||
let rest = rest.trim_start_matches("static ").trim();
|
||||
// `using Alias = Namespace.Type;` — keep only the right-hand path.
|
||||
let rest = rest.split('=').next().unwrap_or(rest).trim();
|
||||
let target = rest.trim_end_matches(';').trim();
|
||||
if !target.is_empty() {
|
||||
out.push(Import {
|
||||
target: target.to_string(),
|
||||
import_type: "using".into(),
|
||||
levels: 0,
|
||||
});
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn extract_java(content: &str) -> Vec<Import> {
|
||||
let mut out = Vec::new();
|
||||
for line in content.lines() {
|
||||
@@ -311,7 +361,7 @@ fn clean(target: &str) -> Option<String> {
|
||||
const MODULE_ENTRIES: &[&str] = &["mod.rs", "__init__.py", "index.ts", "index.js", "index.tsx"];
|
||||
|
||||
impl Resolver {
|
||||
pub fn build(files: Vec<(String, i64)>, crate_name: Option<&str>) -> Self {
|
||||
pub fn build(files: Vec<(String, i64)>, crate_aliases: &[(String, String)]) -> Self {
|
||||
let mut by_path = HashMap::new();
|
||||
let mut by_name = HashMap::new();
|
||||
let mut by_stem = HashMap::new();
|
||||
@@ -325,16 +375,16 @@ impl Resolver {
|
||||
.unwrap_or(name);
|
||||
by_stem.entry(stem).or_insert(id);
|
||||
}
|
||||
// `use <crate>::…` names the crate's own lib root, which always lives
|
||||
// at src/lib.rs — there is no `<crate>.rs` file. Register the package
|
||||
// name as an alias so bin→lib edges resolve. Workspaces (no [package]
|
||||
// name) are skipped.
|
||||
if let Some(name) = crate_name.filter(|n| !n.is_empty())
|
||||
&& let Some(id) = by_path.get("src/lib.rs").copied()
|
||||
{
|
||||
by_path.insert(name.to_string(), id);
|
||||
by_name.entry(name.to_string()).or_insert(id);
|
||||
by_stem.entry(name.to_string()).or_insert(id);
|
||||
// `use <crate>::…` names a crate's lib root, which always lives at
|
||||
// `src/lib.rs` — there is no `<crate>.rs` file. Register each package
|
||||
// name (root crate + workspace members) as an alias for its lib root so
|
||||
// cross-crate edges resolve.
|
||||
for (name, rel) in crate_aliases {
|
||||
if let Some(id) = by_path.get(rel).copied() {
|
||||
by_path.insert(name.clone(), id);
|
||||
by_name.entry(name.clone()).or_insert(id);
|
||||
by_stem.entry(name.clone()).or_insert(id);
|
||||
}
|
||||
}
|
||||
Resolver {
|
||||
by_path,
|
||||
|
||||
+333
-26
@@ -7,6 +7,7 @@ use anyhow::{Context, Result};
|
||||
use rayon::prelude::*;
|
||||
use rusqlite::params;
|
||||
|
||||
use crate::config;
|
||||
use crate::ctags;
|
||||
use crate::discovery::{self, SourceFile};
|
||||
use crate::index::{hashing, imports, store};
|
||||
@@ -18,10 +19,138 @@ pub struct IndexStats {
|
||||
pub imports: i64,
|
||||
}
|
||||
|
||||
/// Read the `[package] name` from `Cargo.toml`, normalized to the crate
|
||||
/// identifier (`-` → `_`) used in `use <crate>::…`.
|
||||
fn read_crate_name(project: &Path) -> Option<String> {
|
||||
let toml = std::fs::read_to_string(project.join("Cargo.toml")).ok()?;
|
||||
/// Read `(crate_name, lib_rel_path)` aliases from `Cargo.toml`: the root
|
||||
/// `[package]` plus every `[workspace] members` entry. Crate names are
|
||||
/// normalized to the identifier (`-` → `_`) used in `use <crate>::…`.
|
||||
fn read_crate_aliases(project: &Path) -> Vec<(String, String)> {
|
||||
let Some(toml) = std::fs::read_to_string(project.join("Cargo.toml")).ok() else {
|
||||
return Vec::new();
|
||||
};
|
||||
let mut aliases = Vec::new();
|
||||
let mut root_name = None;
|
||||
let mut members = Vec::new();
|
||||
let mut section = String::new();
|
||||
for line in toml.lines() {
|
||||
let l = line.trim();
|
||||
if l.starts_with('[') {
|
||||
section = l.trim_matches(['[', ']']).trim().to_string();
|
||||
continue;
|
||||
}
|
||||
if section == "package"
|
||||
&& root_name.is_none()
|
||||
&& let Some(v) = l.strip_prefix("name").map(str::trim)
|
||||
&& let Some(v) = v.strip_prefix('=').map(str::trim)
|
||||
{
|
||||
root_name = Some(v.trim_matches('"').replace('-', "_"));
|
||||
} else if section == "workspace"
|
||||
&& let Some(v) = l.strip_prefix("members").map(str::trim)
|
||||
&& let Some(v) = v.strip_prefix('=').map(str::trim)
|
||||
{
|
||||
members = parse_members(v);
|
||||
}
|
||||
}
|
||||
if let Some(name) = root_name
|
||||
&& !name.is_empty()
|
||||
{
|
||||
aliases.push((name, "src/lib.rs".to_string()));
|
||||
}
|
||||
for m in members {
|
||||
if m.contains('*') || m.contains('?') {
|
||||
for expanded in expand_member_globs(project, &m) {
|
||||
if let Some(name) = member_package_name(project, &expanded) {
|
||||
aliases.push((name, format!("{expanded}/src/lib.rs")));
|
||||
}
|
||||
}
|
||||
} else if let Some(name) = member_package_name(project, &m) {
|
||||
aliases.push((name, format!("{m}/src/lib.rs")));
|
||||
}
|
||||
}
|
||||
aliases
|
||||
}
|
||||
|
||||
/// Extract entries from a `members = ["a", "b"]` value, keeping globs for later
|
||||
/// expansion.
|
||||
fn parse_members(v: &str) -> Vec<String> {
|
||||
let inner = match (v.find('['), v.rfind(']')) {
|
||||
(Some(s), Some(e)) if e > s => &v[s + 1..e],
|
||||
_ => v,
|
||||
};
|
||||
inner
|
||||
.split(',')
|
||||
.map(|p| p.trim().trim_matches('"').trim())
|
||||
.filter(|p| !p.is_empty())
|
||||
.map(str::to_string)
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Expand a `members` glob (e.g. `crates/*`) to member directories that contain
|
||||
/// a `Cargo.toml`. Only the trailing path component may contain a glob.
|
||||
fn expand_member_globs(project: &Path, pattern: &str) -> Vec<String> {
|
||||
let comps: Vec<&str> = pattern.split('/').collect();
|
||||
let Some((last, prefix_comps)) = comps.split_last() else {
|
||||
return Vec::new();
|
||||
};
|
||||
if !last.contains('*') && !last.contains('?') {
|
||||
return Vec::new();
|
||||
}
|
||||
let prefix = prefix_comps.join("/");
|
||||
let base = if prefix.is_empty() {
|
||||
project.to_path_buf()
|
||||
} else {
|
||||
project.join(&prefix)
|
||||
};
|
||||
let Ok(entries) = std::fs::read_dir(&base) else {
|
||||
return Vec::new();
|
||||
};
|
||||
let mut out = Vec::new();
|
||||
for e in entries.flatten() {
|
||||
if !e.file_type().map(|t| t.is_dir()).unwrap_or(false) {
|
||||
continue;
|
||||
}
|
||||
let name = e.file_name().to_string_lossy().to_string();
|
||||
if !glob_matches(last, &name) {
|
||||
continue;
|
||||
}
|
||||
let rel = if prefix.is_empty() {
|
||||
name
|
||||
} else {
|
||||
format!("{prefix}/{name}")
|
||||
};
|
||||
if project.join(&rel).join("Cargo.toml").is_file() {
|
||||
out.push(rel);
|
||||
}
|
||||
}
|
||||
out.sort();
|
||||
out
|
||||
}
|
||||
|
||||
/// Minimal glob matcher for `*` (any sequence) and `?` (single char).
|
||||
fn glob_matches(pattern: &str, text: &str) -> bool {
|
||||
let p: Vec<char> = pattern.chars().collect();
|
||||
let t: Vec<char> = text.chars().collect();
|
||||
let (np, nt) = (p.len(), t.len());
|
||||
let mut dp = vec![vec![false; nt + 1]; np + 1];
|
||||
dp[0][0] = true;
|
||||
for i in 1..=np {
|
||||
if p[i - 1] == '*' {
|
||||
dp[i][0] = dp[i - 1][0];
|
||||
}
|
||||
}
|
||||
for i in 1..=np {
|
||||
for j in 1..=nt {
|
||||
dp[i][j] = match p[i - 1] {
|
||||
'*' => dp[i - 1][j] || dp[i][j - 1],
|
||||
'?' => dp[i - 1][j - 1],
|
||||
c => dp[i - 1][j - 1] && c == t[j - 1],
|
||||
};
|
||||
}
|
||||
}
|
||||
dp[np][nt]
|
||||
}
|
||||
|
||||
/// Read a workspace member's `[package] name`, normalized to a crate identifier.
|
||||
fn member_package_name(project: &Path, member: &str) -> Option<String> {
|
||||
let toml = std::fs::read_to_string(project.join(member).join("Cargo.toml")).ok()?;
|
||||
let mut in_package = false;
|
||||
for line in toml.lines() {
|
||||
let l = line.trim();
|
||||
@@ -62,13 +191,12 @@ pub fn run(project: &Path, db_path: &Path, recreate: bool) -> Result<IndexStats>
|
||||
|
||||
// Hash all files in parallel. Propagate read errors instead of silently
|
||||
// treating an unreadable file as an empty (all-zero) hash.
|
||||
let hashed: Vec<(String, String, i64, &'static str)> = files
|
||||
let hashed: Vec<(String, String, &'static str)> = files
|
||||
.par_iter()
|
||||
.map(|f| {
|
||||
let h = hashing::hash_file(&f.path)
|
||||
.with_context(|| format!("cannot hash {}", f.path.display()))?;
|
||||
let m = hashing::mtime_secs(&f.path);
|
||||
Ok((f.rel.clone(), h, m, f.language))
|
||||
Ok((f.rel.clone(), h, f.language))
|
||||
})
|
||||
.collect::<Result<Vec<_>>>()?;
|
||||
|
||||
@@ -80,7 +208,7 @@ pub fn run(project: &Path, db_path: &Path, recreate: bool) -> Result<IndexStats>
|
||||
|
||||
let changed: Vec<&SourceFile> = hashed
|
||||
.iter()
|
||||
.filter(|(rel, h, _, _)| existing.get(rel).map(|(_, oh)| oh != h).unwrap_or(true))
|
||||
.filter(|(rel, h, _)| existing.get(rel).map(|(_, oh)| oh != h).unwrap_or(true))
|
||||
.filter_map(|(rel, ..)| by_rel.get(rel.as_str()).copied())
|
||||
.collect();
|
||||
|
||||
@@ -93,8 +221,8 @@ pub fn run(project: &Path, db_path: &Path, recreate: bool) -> Result<IndexStats>
|
||||
conn.execute_batch("BEGIN;")?;
|
||||
|
||||
let mut file_ids: HashMap<String, i64> = HashMap::new();
|
||||
for (rel, h, m, lang) in &hashed {
|
||||
let id = store::upsert_file(&conn, rel, h, lang, *m)?;
|
||||
for (rel, h, lang) in &hashed {
|
||||
let id = store::upsert_file(&conn, rel, h, lang)?;
|
||||
file_ids.insert(rel.clone(), id);
|
||||
}
|
||||
|
||||
@@ -115,7 +243,7 @@ pub fn run(project: &Path, db_path: &Path, recreate: bool) -> Result<IndexStats>
|
||||
|
||||
let resolver = imports::Resolver::build(
|
||||
file_ids.iter().map(|(k, v)| (k.clone(), *v)).collect(),
|
||||
read_crate_name(project).as_deref(),
|
||||
&read_crate_aliases(project),
|
||||
);
|
||||
|
||||
// Symbols + content FTS: only files whose content changed. Skipping ctags
|
||||
@@ -137,9 +265,104 @@ pub fn run(project: &Path, db_path: &Path, recreate: bool) -> Result<IndexStats>
|
||||
// change whether an unchanged importer resolves, so resolution must cover
|
||||
// the whole file set, not just the changed set. (Cheap: read + regex, no
|
||||
// ctags subprocess.)
|
||||
for f in &files {
|
||||
reindex_imports(&conn, &files, &file_ids, &resolver)?;
|
||||
|
||||
conn.execute_batch("COMMIT;")?;
|
||||
|
||||
summary(&conn)
|
||||
}
|
||||
|
||||
/// Re-index a single file by path (relative to the project, or absolute under
|
||||
/// it), then re-resolve imports globally. Much faster than `run` for the
|
||||
/// edit-one-file loop agents use: it hashes and runs ctags over one file only.
|
||||
pub fn run_file(project: &Path, db_path: &Path, target: &str) -> Result<IndexStats> {
|
||||
if !db_path.exists() {
|
||||
anyhow::bail!(
|
||||
"no index for {}. Run `rustytokens index create <path>` first.",
|
||||
project.display()
|
||||
);
|
||||
}
|
||||
let conn = store::open(db_path)?;
|
||||
|
||||
let rel = resolve_rel(project, target)?;
|
||||
let abs = project.join(&rel);
|
||||
if !abs.is_file() {
|
||||
anyhow::bail!("not a file: {}", abs.display());
|
||||
}
|
||||
let Some(language) = config::language_for(&abs) else {
|
||||
anyhow::bail!("unsupported file type: {rel}");
|
||||
};
|
||||
|
||||
let hash =
|
||||
hashing::hash_file(&abs).with_context(|| format!("cannot hash {}", abs.display()))?;
|
||||
let tags = ctags::run(&[abs.as_path()])?;
|
||||
let content =
|
||||
std::fs::read_to_string(&abs).with_context(|| format!("cannot read {}", abs.display()))?;
|
||||
let symbols = tags_to_symbols(&tags, &content);
|
||||
|
||||
conn.execute_batch("BEGIN;")?;
|
||||
let fid = store::upsert_file(&conn, &rel, &hash, language)?;
|
||||
store::replace_symbols(&conn, fid, &symbols)?;
|
||||
conn.execute("DELETE FROM content_fts WHERE rowid = ?1", [fid])?;
|
||||
conn.execute(
|
||||
"INSERT INTO content_fts(rowid, path, content) VALUES (?1, ?2, ?3)",
|
||||
params![fid, rel, content],
|
||||
)?;
|
||||
|
||||
// A newly added file can make other files' imports resolve, so re-resolve
|
||||
// the whole import graph (same cheap read+regex pass as `run`).
|
||||
let files = discovery::discover(project)?;
|
||||
let file_ids: HashMap<String, i64> = store::all_files(&conn)?
|
||||
.into_iter()
|
||||
.map(|f| (f.path.clone(), f.id))
|
||||
.collect();
|
||||
let resolver = imports::Resolver::build(
|
||||
file_ids.iter().map(|(k, v)| (k.clone(), *v)).collect(),
|
||||
&read_crate_aliases(project),
|
||||
);
|
||||
reindex_imports(&conn, &files, &file_ids, &resolver)?;
|
||||
|
||||
conn.execute_batch("COMMIT;")?;
|
||||
|
||||
summary(&conn)
|
||||
}
|
||||
|
||||
fn summary(conn: &rusqlite::Connection) -> Result<IndexStats> {
|
||||
let (files_n, symbols_n, imports_n) = store::stats(conn)?;
|
||||
Ok(IndexStats {
|
||||
files: files_n,
|
||||
symbols: symbols_n,
|
||||
imports: imports_n,
|
||||
})
|
||||
}
|
||||
|
||||
/// Resolve a user-supplied file target to a project-relative path. Absolute
|
||||
/// paths under the project are stripped; relative paths pass through unchanged.
|
||||
fn resolve_rel(project: &Path, target: &str) -> Result<String> {
|
||||
let p = Path::new(target);
|
||||
let rel = if p.is_absolute() {
|
||||
let base = project
|
||||
.canonicalize()
|
||||
.unwrap_or_else(|_| project.to_path_buf());
|
||||
p.strip_prefix(&base)
|
||||
.with_context(|| format!("{target} is outside {}", project.display()))?
|
||||
.to_path_buf()
|
||||
} else {
|
||||
p.to_path_buf()
|
||||
};
|
||||
Ok(rel.to_string_lossy().replace('\\', "/"))
|
||||
}
|
||||
|
||||
/// Clear and re-extract imports for every file, resolving against `file_ids`.
|
||||
fn reindex_imports(
|
||||
conn: &rusqlite::Connection,
|
||||
files: &[SourceFile],
|
||||
file_ids: &HashMap<String, i64>,
|
||||
resolver: &imports::Resolver,
|
||||
) -> Result<()> {
|
||||
for f in files {
|
||||
let fid = file_ids[&f.rel];
|
||||
store::clear_imports_for_file(&conn, fid)?;
|
||||
store::clear_imports_for_file(conn, fid)?;
|
||||
let dir = f
|
||||
.rel
|
||||
.rsplit_once('/')
|
||||
@@ -150,19 +373,11 @@ pub fn run(project: &Path, db_path: &Path, recreate: bool) -> Result<IndexStats>
|
||||
if let Some(to) = resolver.resolve_with_context(&imp.target, &dir, imp.levels)
|
||||
&& to != fid
|
||||
{
|
||||
store::insert_import(&conn, fid, to, &imp.import_type)?;
|
||||
store::insert_import(conn, fid, to, &imp.import_type)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
conn.execute_batch("COMMIT;")?;
|
||||
|
||||
let (files_n, symbols_n, imports_n) = store::stats(&conn)?;
|
||||
Ok(IndexStats {
|
||||
files: files_n,
|
||||
symbols: symbols_n,
|
||||
imports: imports_n,
|
||||
})
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn tags_to_symbols(tags: &[ctags::Tag], content: &str) -> Vec<store::NewSymbol> {
|
||||
@@ -206,14 +421,19 @@ fn extend_block_end(lines: &[&str], start: i64, end: i64, kind: &str) -> i64 {
|
||||
let mut depth = 0i64;
|
||||
let mut opened = false;
|
||||
let mut out = end;
|
||||
// ponytail: naive char brace count; braces in string/comment literals skew
|
||||
// depth. Swap for a tokenizer only if real spans come back wrong.
|
||||
let mut st = ScanState::default();
|
||||
for (i, line) in lines.iter().enumerate() {
|
||||
let n = (i as i64) + 1;
|
||||
// Track string/comment state from the top of the file so a literal
|
||||
// opened before `start` doesn't skew the brace count.
|
||||
let mask = code_mask(line, &mut st);
|
||||
if n < start {
|
||||
continue;
|
||||
}
|
||||
for c in line.chars() {
|
||||
for (c, is_code) in line.chars().zip(mask.iter()) {
|
||||
if !is_code {
|
||||
continue;
|
||||
}
|
||||
match c {
|
||||
'{' => {
|
||||
depth += 1;
|
||||
@@ -233,6 +453,93 @@ fn extend_block_end(lines: &[&str], start: i64, end: i64, kind: &str) -> i64 {
|
||||
out
|
||||
}
|
||||
|
||||
/// Lexer state carried across lines: a `/* … */` block comment or a multi-line
|
||||
/// string literal that began on an earlier line.
|
||||
#[derive(Default)]
|
||||
struct ScanState {
|
||||
in_block_comment: bool,
|
||||
/// (quote char, terminator length 1 or 3 for triple-quoted strings)
|
||||
in_string: Option<(char, u8)>,
|
||||
}
|
||||
|
||||
/// Per-char "is code" mask: string literals and comments are masked out so
|
||||
/// brace counting in [`extend_block_end`] ignores braces in literals.
|
||||
fn code_mask(line: &str, st: &mut ScanState) -> Vec<bool> {
|
||||
let chars: Vec<char> = line.chars().collect();
|
||||
let n = chars.len();
|
||||
let mut mask = vec![true; n];
|
||||
let mut i = 0;
|
||||
while i < n {
|
||||
if st.in_block_comment {
|
||||
mask[i] = false;
|
||||
if chars[i] == '*' && i + 1 < n && chars[i + 1] == '/' {
|
||||
mask[i + 1] = false;
|
||||
st.in_block_comment = false;
|
||||
i += 2;
|
||||
continue;
|
||||
}
|
||||
i += 1;
|
||||
continue;
|
||||
}
|
||||
if let Some((q, len)) = st.in_string {
|
||||
let term = len as usize;
|
||||
if i + term <= n && chars[i..i + term].iter().all(|&c| c == q) {
|
||||
for k in 0..term {
|
||||
mask[i + k] = false;
|
||||
}
|
||||
st.in_string = None;
|
||||
i += term;
|
||||
} else if term == 1 && chars[i] == '\\' && i + 1 < n {
|
||||
mask[i] = false;
|
||||
mask[i + 1] = false;
|
||||
i += 2;
|
||||
} else {
|
||||
mask[i] = false;
|
||||
i += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
let c = chars[i];
|
||||
if c == '/' && i + 1 < n {
|
||||
if chars[i + 1] == '*' {
|
||||
mask[i] = false;
|
||||
mask[i + 1] = false;
|
||||
st.in_block_comment = true;
|
||||
i += 2;
|
||||
continue;
|
||||
}
|
||||
if chars[i + 1] == '/' {
|
||||
mask[i..].fill(false);
|
||||
break;
|
||||
}
|
||||
}
|
||||
if c == '#' {
|
||||
let prev_ws = i == 0 || chars[i - 1].is_whitespace();
|
||||
let next = chars.get(i + 1).copied();
|
||||
if prev_ws && next != Some('[') && next != Some('!') {
|
||||
mask[i..].fill(false);
|
||||
break;
|
||||
}
|
||||
}
|
||||
if c == '"' || c == '\'' || c == '`' {
|
||||
let triple = i + 2 < n && chars[i + 1] == c && chars[i + 2] == c;
|
||||
mask[i] = false;
|
||||
if triple {
|
||||
mask[i + 1] = false;
|
||||
mask[i + 2] = false;
|
||||
st.in_string = Some((c, 3));
|
||||
i += 3;
|
||||
} else {
|
||||
st.in_string = Some((c, 1));
|
||||
i += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
i += 1;
|
||||
}
|
||||
mask
|
||||
}
|
||||
|
||||
fn is_block_kind(kind: &str) -> bool {
|
||||
matches!(
|
||||
kind,
|
||||
|
||||
+39
-18
@@ -1,19 +1,21 @@
|
||||
//! SQLite schema + minimal migration (schema_version in metadata).
|
||||
//! SQLite schema + migration (schema_version in metadata).
|
||||
|
||||
use anyhow::Result;
|
||||
use anyhow::{Context, Result};
|
||||
use rusqlite::Connection;
|
||||
|
||||
const SCHEMA_VERSION: i64 = 1;
|
||||
const SCHEMA_VERSION: i64 = 2;
|
||||
|
||||
/// Fresh-install schema at the current version. Everything is `IF NOT EXISTS`
|
||||
/// so it can be re-run safely (the old version re-ran it on every `version <
|
||||
/// SCHEMA_VERSION`, which broke the moment any table already existed).
|
||||
const SCHEMA: &str = r#"
|
||||
CREATE TABLE files (
|
||||
CREATE TABLE IF NOT EXISTS files (
|
||||
id INTEGER PRIMARY KEY,
|
||||
path TEXT NOT NULL UNIQUE,
|
||||
content_hash TEXT NOT NULL,
|
||||
language TEXT NOT NULL,
|
||||
mtime INTEGER NOT NULL
|
||||
language TEXT NOT NULL
|
||||
);
|
||||
CREATE TABLE symbols (
|
||||
CREATE TABLE IF NOT EXISTS symbols (
|
||||
id INTEGER PRIMARY KEY,
|
||||
file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL,
|
||||
@@ -23,31 +25,40 @@ CREATE TABLE symbols (
|
||||
line_start INTEGER NOT NULL,
|
||||
line_end INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX idx_symbols_file ON symbols(file_id);
|
||||
CREATE INDEX idx_symbols_name ON symbols(name);
|
||||
CREATE TABLE imports (
|
||||
CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(file_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_symbols_name ON symbols(name);
|
||||
CREATE TABLE IF NOT EXISTS imports (
|
||||
id INTEGER PRIMARY KEY,
|
||||
from_file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
|
||||
to_file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
|
||||
import_type TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX idx_imports_from ON imports(from_file_id);
|
||||
CREATE INDEX idx_imports_to ON imports(to_file_id);
|
||||
CREATE VIRTUAL TABLE symbols_fts USING fts5(
|
||||
CREATE INDEX IF NOT EXISTS idx_imports_from ON imports(from_file_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_imports_to ON imports(to_file_id);
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS symbols_fts USING fts5(
|
||||
name, qualified_name, kind, signature,
|
||||
content='symbols', content_rowid='id'
|
||||
);
|
||||
CREATE TRIGGER symbols_ai AFTER INSERT ON symbols BEGIN
|
||||
CREATE TRIGGER IF NOT EXISTS symbols_ai AFTER INSERT ON symbols BEGIN
|
||||
INSERT INTO symbols_fts(rowid, name, qualified_name, kind, signature)
|
||||
VALUES (new.id, new.name, new.qualified_name, new.kind, new.signature);
|
||||
END;
|
||||
CREATE TRIGGER symbols_ad AFTER DELETE ON symbols BEGIN
|
||||
CREATE TRIGGER IF NOT EXISTS symbols_ad AFTER DELETE ON symbols BEGIN
|
||||
INSERT INTO symbols_fts(symbols_fts, rowid, name, qualified_name, kind, signature)
|
||||
VALUES ('delete', old.id, old.name, old.qualified_name, old.kind, old.signature);
|
||||
END;
|
||||
CREATE VIRTUAL TABLE content_fts USING fts5(path UNINDEXED, content);
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS content_fts USING fts5(path UNINDEXED, content);
|
||||
"#;
|
||||
|
||||
/// Incremental migrations. `MIGRATIONS[i]` upgrades from schema version `i+1`
|
||||
/// to `i+2`; version 0 is a fresh database and is created via [`SCHEMA`]
|
||||
/// directly.
|
||||
const MIGRATIONS: &[&str] = &[
|
||||
// v1 -> v2: the `mtime` column was written but never read (change detection
|
||||
// is content-hash based). Drop the dead column.
|
||||
"ALTER TABLE files DROP COLUMN mtime;",
|
||||
];
|
||||
|
||||
/// Create schema if missing, or migrate to the current version.
|
||||
pub fn init(conn: &mut Connection) -> Result<()> {
|
||||
conn.execute_batch(
|
||||
@@ -60,12 +71,22 @@ pub fn init(conn: &mut Connection) -> Result<()> {
|
||||
|r| r.get(0),
|
||||
)
|
||||
.unwrap_or(0);
|
||||
if version < SCHEMA_VERSION {
|
||||
|
||||
if version == 0 {
|
||||
conn.execute_batch(SCHEMA)?;
|
||||
} else {
|
||||
for (i, migration) in MIGRATIONS.iter().enumerate() {
|
||||
// i-th migration upgrades version (i+1) -> (i+2).
|
||||
if version <= (i as i64) + 1 {
|
||||
conn.execute_batch(migration)
|
||||
.with_context(|| format!("migration to schema v{} failed", i + 2))?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO metadata (key, value) VALUES ('schema_version', ?1)",
|
||||
[SCHEMA_VERSION.to_string()],
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
+51
-15
@@ -69,17 +69,11 @@ pub fn all_files(conn: &Connection) -> Result<Vec<FileRow>> {
|
||||
Ok(rows.collect::<std::result::Result<Vec<_>, _>>()?)
|
||||
}
|
||||
|
||||
pub fn upsert_file(
|
||||
conn: &Connection,
|
||||
path: &str,
|
||||
hash: &str,
|
||||
language: &str,
|
||||
mtime: i64,
|
||||
) -> Result<i64> {
|
||||
pub fn upsert_file(conn: &Connection, path: &str, hash: &str, language: &str) -> Result<i64> {
|
||||
conn.execute(
|
||||
"INSERT INTO files (path, content_hash, language, mtime) VALUES (?1, ?2, ?3, ?4)
|
||||
ON CONFLICT(path) DO UPDATE SET content_hash=?2, language=?3, mtime=?4",
|
||||
params![path, hash, language, mtime],
|
||||
"INSERT INTO files (path, content_hash, language) VALUES (?1, ?2, ?3)
|
||||
ON CONFLICT(path) DO UPDATE SET content_hash=?2, language=?3",
|
||||
params![path, hash, language],
|
||||
)?;
|
||||
// last_insert_rowid() is unreliable after an upsert that updates; query it.
|
||||
Ok(conn.query_row("SELECT id FROM files WHERE path = ?1", [path], |r| r.get(0))?)
|
||||
@@ -179,6 +173,27 @@ pub fn symbol_by_id(conn: &Connection, id: i64) -> Result<Option<SymbolRow>> {
|
||||
.map_err(Into::into)
|
||||
}
|
||||
|
||||
/// All symbols whose name equals `name` (case-insensitive), ordered by id.
|
||||
pub fn symbols_by_name(conn: &Connection, name: &str) -> Result<Vec<SymbolRow>> {
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT id, file_id, name, qualified_name, kind, signature, line_start, line_end
|
||||
FROM symbols WHERE name = ?1 COLLATE NOCASE ORDER BY id",
|
||||
)?;
|
||||
let rows = stmt.query_map([name], |r| {
|
||||
Ok(SymbolRow {
|
||||
id: r.get(0)?,
|
||||
file_id: r.get(1)?,
|
||||
name: r.get(2)?,
|
||||
qualified_name: r.get(3)?,
|
||||
kind: r.get(4)?,
|
||||
signature: r.get(5)?,
|
||||
line_start: r.get(6)?,
|
||||
line_end: r.get(7)?,
|
||||
})
|
||||
})?;
|
||||
Ok(rows.collect::<std::result::Result<Vec<_>, _>>()?)
|
||||
}
|
||||
|
||||
pub fn symbols_for_file(conn: &Connection, file_id: i64) -> Result<Vec<SymbolRow>> {
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT id, file_id, name, qualified_name, kind, signature, line_start, line_end
|
||||
@@ -199,6 +214,27 @@ pub fn symbols_for_file(conn: &Connection, file_id: i64) -> Result<Vec<SymbolRow
|
||||
Ok(rows.collect::<std::result::Result<Vec<_>, _>>()?)
|
||||
}
|
||||
|
||||
/// Every symbol in the index, ordered by id.
|
||||
pub fn all_symbols(conn: &Connection) -> Result<Vec<SymbolRow>> {
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT id, file_id, name, qualified_name, kind, signature, line_start, line_end
|
||||
FROM symbols ORDER BY id",
|
||||
)?;
|
||||
let rows = stmt.query_map([], |r| {
|
||||
Ok(SymbolRow {
|
||||
id: r.get(0)?,
|
||||
file_id: r.get(1)?,
|
||||
name: r.get(2)?,
|
||||
qualified_name: r.get(3)?,
|
||||
kind: r.get(4)?,
|
||||
signature: r.get(5)?,
|
||||
line_start: r.get(6)?,
|
||||
line_end: r.get(7)?,
|
||||
})
|
||||
})?;
|
||||
Ok(rows.collect::<std::result::Result<Vec<_>, _>>()?)
|
||||
}
|
||||
|
||||
pub fn importers_of(conn: &Connection, file_id: i64) -> Result<Vec<i64>> {
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT DISTINCT from_file_id FROM imports WHERE to_file_id = ?1 ORDER BY from_file_id",
|
||||
@@ -236,8 +272,8 @@ mod tests {
|
||||
#[test]
|
||||
fn file_insert_query_and_upsert() {
|
||||
let conn = mem();
|
||||
let id1 = upsert_file(&conn, "src/lib.rs", "hash1", "rust", 100).unwrap();
|
||||
let id2 = upsert_file(&conn, "src/lib.rs", "hash2", "rust", 200).unwrap();
|
||||
let id1 = upsert_file(&conn, "src/lib.rs", "hash1", "rust").unwrap();
|
||||
let id2 = upsert_file(&conn, "src/lib.rs", "hash2", "rust").unwrap();
|
||||
assert_eq!(id1, id2, "upsert reuses the same row");
|
||||
let row = file_by_id(&conn, id1).unwrap().unwrap();
|
||||
assert_eq!(row.content_hash, "hash2");
|
||||
@@ -248,7 +284,7 @@ mod tests {
|
||||
#[test]
|
||||
fn symbols_replace_and_query() {
|
||||
let conn = mem();
|
||||
let fid = upsert_file(&conn, "src/lib.rs", "h", "rust", 0).unwrap();
|
||||
let fid = upsert_file(&conn, "src/lib.rs", "h", "rust").unwrap();
|
||||
replace_symbols(
|
||||
&conn,
|
||||
fid,
|
||||
@@ -301,8 +337,8 @@ mod tests {
|
||||
#[test]
|
||||
fn imports_roundtrip_and_importers() {
|
||||
let conn = mem();
|
||||
let a = upsert_file(&conn, "a.ts", "h", "typescript", 0).unwrap();
|
||||
let b = upsert_file(&conn, "b.ts", "h", "typescript", 0).unwrap();
|
||||
let a = upsert_file(&conn, "a.ts", "h", "typescript").unwrap();
|
||||
let b = upsert_file(&conn, "b.ts", "h", "typescript").unwrap();
|
||||
insert_import(&conn, a, b, "import").unwrap();
|
||||
let imports = all_imports(&conn).unwrap();
|
||||
assert_eq!(imports.len(), 1);
|
||||
|
||||
+21
-4
@@ -33,15 +33,25 @@ fn run() -> Result<()> {
|
||||
Command::Index { sub } => match sub {
|
||||
IndexCommand::Create { path } => emit(&ops::index_create(path)?, &cli)?,
|
||||
IndexCommand::Update { path } => emit(&ops::index_update(path)?, &cli)?,
|
||||
IndexCommand::File { file } => emit(&ops::index_file(&cli.project, file)?, &cli)?,
|
||||
},
|
||||
Command::Search { sub } => match sub {
|
||||
SearchCommand::Symbols { query } => {
|
||||
emit(&ops::search_symbols(&cli.project, query)?, &cli)?
|
||||
SearchCommand::Symbols {
|
||||
query,
|
||||
include_source,
|
||||
limit,
|
||||
} => emit(
|
||||
&ops::search_symbols(&cli.project, query, *include_source, *limit)?,
|
||||
&cli,
|
||||
)?,
|
||||
SearchCommand::Text { query, limit } => {
|
||||
emit(&ops::search_text(&cli.project, query, *limit)?, &cli)?
|
||||
}
|
||||
SearchCommand::Text { query } => emit(&ops::search_text(&cli.project, query)?, &cli)?,
|
||||
},
|
||||
Command::Inspect { sub } => match sub {
|
||||
InspectCommand::Symbol { id } => emit(&ops::inspect_symbol(&cli.project, *id)?, &cli)?,
|
||||
InspectCommand::Symbol { target } => {
|
||||
emit(&ops::inspect_symbol(&cli.project, target)?, &cli)?
|
||||
}
|
||||
InspectCommand::Outline { file } => {
|
||||
emit(&ops::inspect_outline(&cli.project, file)?, &cli)?
|
||||
}
|
||||
@@ -63,6 +73,13 @@ fn run() -> Result<()> {
|
||||
emit(&ops::find_importers(&cli.project, file)?, &cli)?
|
||||
}
|
||||
FindCommand::Dead => emit(&ops::find_dead(&cli.project)?, &cli)?,
|
||||
FindCommand::DeadSymbols => emit(&ops::find_dead_symbols(&cli.project)?, &cli)?,
|
||||
FindCommand::Callers { target } => {
|
||||
emit(&ops::find_callers(&cli.project, target)?, &cli)?
|
||||
}
|
||||
FindCommand::References { identifier } => {
|
||||
emit(&ops::find_references(&cli.project, identifier)?, &cli)?
|
||||
}
|
||||
},
|
||||
Command::Serve => mcp::serve(&cli.project)?,
|
||||
Command::Stats => emit(&ops::stats(&cli.project)?, &cli)?,
|
||||
|
||||
+107
-33
@@ -123,20 +123,39 @@ fn dispatch(name: &str, args: &Value, project: &Path) -> Result<Value> {
|
||||
.filter(|s| !s.is_empty())
|
||||
.ok_or_else(|| anyhow::anyhow!("{tool}: missing or invalid argument '{key}'"))
|
||||
}
|
||||
fn req_i64(args: &Value, key: &str, tool: &str) -> Result<i64> {
|
||||
args.get(key)
|
||||
.and_then(|v| v.as_i64())
|
||||
.ok_or_else(|| anyhow::anyhow!("{tool}: missing or invalid argument '{key}'"))
|
||||
}
|
||||
let opt_str = |k: &str| args.get(k).and_then(|v| v.as_str());
|
||||
let opt_depth = |k: &str| args.get(k).and_then(|v| v.as_u64()).map(|d| d as usize);
|
||||
|
||||
match name {
|
||||
"index_create" => ops::index_create(Path::new(req_str(args, "path", name)?)),
|
||||
"index_update" => ops::index_update(Path::new(req_str(args, "path", name)?)),
|
||||
"search_symbols" => ops::search_symbols(project, req_str(args, "query", name)?),
|
||||
"search_text" => ops::search_text(project, req_str(args, "query", name)?),
|
||||
"inspect_symbol" => ops::inspect_symbol(project, req_i64(args, "id", name)?),
|
||||
"index_file" => ops::index_file(project, req_str(args, "path", name)?),
|
||||
"search_symbols" => ops::search_symbols(
|
||||
project,
|
||||
req_str(args, "query", name)?,
|
||||
args.get("include_source")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false),
|
||||
args.get("limit")
|
||||
.and_then(|v| v.as_u64())
|
||||
.map(|n| n as usize)
|
||||
.unwrap_or(200),
|
||||
),
|
||||
"search_text" => ops::search_text(
|
||||
project,
|
||||
req_str(args, "query", name)?,
|
||||
args.get("limit")
|
||||
.and_then(|v| v.as_u64())
|
||||
.map(|n| n as usize)
|
||||
.unwrap_or(100),
|
||||
),
|
||||
"inspect_symbol" => {
|
||||
if let Some(id) = args.get("id").and_then(|v| v.as_i64()) {
|
||||
ops::inspect_symbol(project, &id.to_string())
|
||||
} else {
|
||||
ops::inspect_symbol(project, req_str(args, "name", name)?)
|
||||
}
|
||||
}
|
||||
"inspect_outline" => ops::inspect_outline(project, req_str(args, "file", name)?),
|
||||
"inspect_file" => {
|
||||
ops::inspect_file(project, req_str(args, "file", name)?, opt_str("lines"))
|
||||
@@ -144,6 +163,15 @@ fn dispatch(name: &str, args: &Value, project: &Path) -> Result<Value> {
|
||||
"inspect_tree" => ops::inspect_tree(project, opt_depth("depth")),
|
||||
"find_importers" => ops::find_importers(project, req_str(args, "file", name)?),
|
||||
"find_dead" => ops::find_dead(project),
|
||||
"find_dead_symbols" => ops::find_dead_symbols(project),
|
||||
"find_callers" => {
|
||||
if let Some(id) = args.get("id").and_then(|v| v.as_i64()) {
|
||||
ops::find_callers(project, &id.to_string())
|
||||
} else {
|
||||
ops::find_callers(project, req_str(args, "name", name)?)
|
||||
}
|
||||
}
|
||||
"find_references" => ops::find_references(project, req_str(args, "identifier", name)?),
|
||||
"inspect_cycles" => ops::inspect_cycles(project),
|
||||
"inspect_blast_radius" => {
|
||||
ops::inspect_blast_radius(project, req_str(args, "file", name)?, opt_depth("depth"))
|
||||
@@ -157,7 +185,7 @@ fn tools() -> Value {
|
||||
json!([
|
||||
tool(
|
||||
"index_create",
|
||||
"Index a project from scratch",
|
||||
"Index a project from scratch: scan source files, extract symbols via ctags and imports, and write a SQLite index. Run before any other tool on a new project, or to rebuild a stale index.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": { "path": { "type": "string", "description": "Project directory to index" } },
|
||||
@@ -166,103 +194,149 @@ fn tools() -> Value {
|
||||
),
|
||||
tool(
|
||||
"index_update",
|
||||
"Re-index only changed files",
|
||||
"Re-index only files whose content hash changed since the last index. Faster than index_create for incremental refreshes after edits.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": { "path": { "type": "string" } },
|
||||
"properties": { "path": { "type": "string", "description": "Project directory to refresh" } },
|
||||
"required": ["path"]
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"index_file",
|
||||
"Re-index a single file (path relative to the indexed project, or absolute). Use after editing one file to refresh its symbols and imports without a full index_update.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": { "path": { "type": "string", "description": "File to re-index, relative to the indexed project (or absolute)" } },
|
||||
"required": ["path"]
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"search_symbols",
|
||||
"Search symbols by name",
|
||||
"Search symbols (functions, classes, methods, structs, etc.) by name. Matches exact, prefix, then full-text; each hit carries id, kind, file, and line range. Set include_source to fetch the symbol's source inline, avoiding a separate inspect_symbol call.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": { "query": { "type": "string" } },
|
||||
"properties": {
|
||||
"query": { "type": "string", "description": "Symbol name or partial name to search" },
|
||||
"include_source": {
|
||||
"type": "boolean",
|
||||
"description": "Include each symbol's source inline (default false)"
|
||||
},
|
||||
"limit": { "type": "integer", "description": "Maximum results (default 200)" }
|
||||
},
|
||||
"required": ["query"]
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"search_text",
|
||||
"Full-text search over file contents",
|
||||
"Full-text search over file contents. Returns file, line number, and a snippet for every matching line. Use to find string literals, comments, or code fragments that are not indexed as symbols.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": { "query": { "type": "string" } },
|
||||
"properties": {
|
||||
"query": { "type": "string", "description": "Text or phrase to search for" },
|
||||
"limit": { "type": "integer", "description": "Maximum matching files (default 100)" }
|
||||
},
|
||||
"required": ["query"]
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"inspect_symbol",
|
||||
"Get the source code of a single symbol by id",
|
||||
"Get the full source code of a symbol by id (from search_symbols) or exact name. Names may be ambiguous; an id always resolves to one symbol.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": { "id": { "type": "integer" } },
|
||||
"required": ["id"]
|
||||
"properties": {
|
||||
"id": { "type": "integer", "description": "Symbol id from search_symbols (unambiguous)" },
|
||||
"name": { "type": "string", "description": "Exact symbol name (alternative to id; may match multiple)" }
|
||||
}
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"inspect_outline",
|
||||
"Get the symbol outline of a file",
|
||||
"Get the symbol outline of a file: names, kinds, signatures, and line ranges for every top-level symbol. Use to orient yourself in a file without reading it whole.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": { "file": { "type": "string" } },
|
||||
"properties": { "file": { "type": "string", "description": "Path relative to the indexed project" } },
|
||||
"required": ["file"]
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"inspect_file",
|
||||
"Read a file or a line range",
|
||||
"Read a file or a line range from the indexed project. Prefer this over reading whole files.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"file": { "type": "string" },
|
||||
"lines": { "type": "string", "description": "Optional range, e.g. 10-40" }
|
||||
"file": { "type": "string", "description": "Path relative to the indexed project" },
|
||||
"lines": { "type": "string", "description": "Optional 1-based inclusive range, e.g. 10-40" }
|
||||
},
|
||||
"required": ["file"]
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"inspect_tree",
|
||||
"Show the indexed file tree",
|
||||
"Show the indexed file tree (directories and files). Use to understand project layout.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": { "depth": { "type": "integer" } }
|
||||
"properties": { "depth": { "type": "integer", "description": "Maximum tree depth" } }
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"find_importers",
|
||||
"List files that import the given file",
|
||||
"List files that import the given file. Use to see who depends on a module before changing it.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": { "file": { "type": "string" } },
|
||||
"properties": { "file": { "type": "string", "description": "Path relative to the indexed project" } },
|
||||
"required": ["file"]
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"find_dead",
|
||||
"List files with no importers",
|
||||
"List files with no importers (entry-point files like main/lib are excluded). Use to spot orphaned or unreferenced modules.",
|
||||
json!({ "type": "object" })
|
||||
),
|
||||
tool(
|
||||
"find_dead_symbols",
|
||||
"List symbols never referenced by name outside their own definition. Heuristic: framework-wired symbols (routes, test functions) may appear here because they are invoked by convention, not by name. Use to find genuinely unused functions and methods.",
|
||||
json!({ "type": "object" })
|
||||
),
|
||||
tool(
|
||||
"find_callers",
|
||||
"Find symbols that call the given symbol (by id or exact name). Searches the symbol's own file plus transitively dependent files.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "integer", "description": "Symbol id from search_symbols" },
|
||||
"name": { "type": "string", "description": "Exact symbol name (alternative to id)" }
|
||||
}
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"find_references",
|
||||
"Find every line where an identifier occurs as a whole word, across indexed files. Use for rename impact analysis.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": { "identifier": { "type": "string", "description": "Identifier or module name to search for" } },
|
||||
"required": ["identifier"]
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"inspect_cycles",
|
||||
"Find circular imports",
|
||||
"Find circular imports (dependency cycles) between files. Use to diagnose build or import problems.",
|
||||
json!({ "type": "object" })
|
||||
),
|
||||
tool(
|
||||
"inspect_blast_radius",
|
||||
"Files transitively affected by a change",
|
||||
"List files transitively affected by a change to a given file (reverse dependency graph). Use before refactoring to gauge impact.",
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"file": { "type": "string" },
|
||||
"depth": { "type": "integer" }
|
||||
"file": { "type": "string", "description": "Path relative to the indexed project" },
|
||||
"depth": { "type": "integer", "description": "Maximum traversal depth" }
|
||||
},
|
||||
"required": ["file"]
|
||||
})
|
||||
),
|
||||
tool(
|
||||
"stats",
|
||||
"Show index statistics",
|
||||
"Show index statistics: file, symbol, and import counts, plus language breakdown.",
|
||||
json!({ "type": "object" })
|
||||
),
|
||||
])
|
||||
|
||||
+555
-44
@@ -1,6 +1,7 @@
|
||||
//! High-level operations shared by the CLI and the MCP server.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::collections::{BTreeMap, HashMap, HashSet};
|
||||
use std::io::BufRead;
|
||||
use std::path::Path;
|
||||
|
||||
use anyhow::{Context, Result, bail};
|
||||
@@ -27,24 +28,113 @@ pub fn index_update(project: &Path) -> Result<Value> {
|
||||
Ok(serde_json::to_value(stats)?)
|
||||
}
|
||||
|
||||
// ---- search ----
|
||||
|
||||
pub fn search_symbols(project: &Path, query: &str) -> Result<Value> {
|
||||
let conn = open_read(project)?;
|
||||
Ok(serde_json::to_value(search::search_symbols(&conn, query)?)?)
|
||||
pub fn index_file(project: &Path, file: &str) -> Result<Value> {
|
||||
let db = config::index_db_path(project);
|
||||
let stats = pipeline::run_file(project, &db, file)?;
|
||||
Ok(serde_json::to_value(stats)?)
|
||||
}
|
||||
|
||||
pub fn search_text(project: &Path, query: &str) -> Result<Value> {
|
||||
// ---- search ----
|
||||
|
||||
pub fn search_symbols(
|
||||
project: &Path,
|
||||
query: &str,
|
||||
include_source: bool,
|
||||
limit: usize,
|
||||
) -> Result<Value> {
|
||||
let conn = open_read(project)?;
|
||||
Ok(serde_json::to_value(search::search_text(&conn, query)?)?)
|
||||
let hits = search::search_symbols(&conn, query, limit)?;
|
||||
if !include_source {
|
||||
return Ok(serde_json::to_value(hits)?);
|
||||
}
|
||||
let values: Vec<Value> = hits
|
||||
.iter()
|
||||
.map(|hit| {
|
||||
let source = read_lines(project, &hit.file, hit.line_start, hit.line_end)?;
|
||||
Ok(serde_json::json!({
|
||||
"id": hit.id,
|
||||
"name": hit.name,
|
||||
"qualified_name": hit.qualified_name,
|
||||
"kind": hit.kind,
|
||||
"signature": hit.signature,
|
||||
"file": hit.file,
|
||||
"line_start": hit.line_start,
|
||||
"line_end": hit.line_end,
|
||||
"match_type": hit.match_type,
|
||||
"source": source,
|
||||
}))
|
||||
})
|
||||
.collect::<Result<_>>()?;
|
||||
Ok(serde_json::json!(values))
|
||||
}
|
||||
|
||||
/// Full-text search over the live contents of indexed files.
|
||||
///
|
||||
/// Reads files from disk (not the `content_fts` snapshot) so edits are visible
|
||||
/// immediately, consistent with `inspect file`/`inspect symbol`. `limit` caps
|
||||
/// the number of files that contribute matches; within a file, identical
|
||||
/// snippets are collapsed to the first occurrence to reduce noise.
|
||||
pub fn search_text(project: &Path, query: &str, limit: usize) -> Result<Value> {
|
||||
let conn = open_read(project)?;
|
||||
let q = query.trim();
|
||||
if q.is_empty() {
|
||||
return Ok(serde_json::json!([]));
|
||||
}
|
||||
let files = store::all_files(&conn)?;
|
||||
let mut out: Vec<search::TextHit> = Vec::new();
|
||||
let mut files_hit = 0usize;
|
||||
for f in files {
|
||||
if files_hit >= limit {
|
||||
break;
|
||||
}
|
||||
let path = project.join(&f.path);
|
||||
let Ok(content) = std::fs::read_to_string(&path) else {
|
||||
continue;
|
||||
};
|
||||
// Collapse identical snippets within a file to the first occurrence.
|
||||
let mut seen: HashSet<String> = HashSet::new();
|
||||
let mut contributed = false;
|
||||
for (line, snippet) in search::lines_matching(&content, q) {
|
||||
if !seen.insert(snippet.clone()) {
|
||||
continue;
|
||||
}
|
||||
out.push(search::TextHit {
|
||||
file: f.path.clone(),
|
||||
line,
|
||||
snippet,
|
||||
});
|
||||
contributed = true;
|
||||
}
|
||||
if contributed {
|
||||
files_hit += 1;
|
||||
}
|
||||
}
|
||||
Ok(serde_json::to_value(out)?)
|
||||
}
|
||||
|
||||
// ---- inspect ----
|
||||
|
||||
pub fn inspect_symbol(project: &Path, id: i64) -> Result<Value> {
|
||||
/// `target` is either a numeric id or a symbol name (exact, case-insensitive).
|
||||
/// Id → single object; name → array of all exact-name matches (full source).
|
||||
pub fn inspect_symbol(project: &Path, target: &str) -> Result<Value> {
|
||||
let conn = open_read(project)?;
|
||||
let sym = store::symbol_by_id(&conn, id)?.with_context(|| format!("no symbol with id {id}"))?;
|
||||
let file = store::file_by_id(&conn, sym.file_id)?
|
||||
let syms = resolve_symbols(&conn, target)?;
|
||||
let objects: Vec<Value> = syms
|
||||
.iter()
|
||||
.map(|sym| symbol_object(project, &conn, sym))
|
||||
.collect::<Result<_>>()?;
|
||||
if target.trim().parse::<i64>().is_ok() {
|
||||
Ok(objects
|
||||
.into_iter()
|
||||
.next()
|
||||
.expect("resolve_symbols is non-empty"))
|
||||
} else {
|
||||
Ok(serde_json::json!(objects))
|
||||
}
|
||||
}
|
||||
|
||||
fn symbol_object(project: &Path, conn: &Connection, sym: &store::SymbolRow) -> Result<Value> {
|
||||
let file = store::file_by_id(conn, sym.file_id)?
|
||||
.with_context(|| format!("no file with id {}", sym.file_id))?;
|
||||
let source = read_lines(project, &file.path, sym.line_start, sym.line_end)?;
|
||||
Ok(serde_json::json!({
|
||||
@@ -60,6 +150,23 @@ pub fn inspect_symbol(project: &Path, id: i64) -> Result<Value> {
|
||||
}))
|
||||
}
|
||||
|
||||
/// Resolve a CLI/MCP `target` to one or more symbols: numeric → id lookup,
|
||||
/// otherwise exact (case-insensitive) name match.
|
||||
fn resolve_symbols(conn: &Connection, target: &str) -> Result<Vec<store::SymbolRow>> {
|
||||
if let Ok(id) = target.trim().parse::<i64>() {
|
||||
let sym =
|
||||
store::symbol_by_id(conn, id)?.with_context(|| format!("no symbol with id {id}"))?;
|
||||
Ok(vec![sym])
|
||||
} else {
|
||||
let name = target.trim();
|
||||
let syms = store::symbols_by_name(conn, name)?;
|
||||
if syms.is_empty() {
|
||||
bail!("no symbol named {name:?}");
|
||||
}
|
||||
Ok(syms)
|
||||
}
|
||||
}
|
||||
|
||||
pub fn inspect_outline(project: &Path, file: &str) -> Result<Value> {
|
||||
let conn = open_read(project)?;
|
||||
let fid = file_id(&conn, file)?;
|
||||
@@ -70,28 +177,20 @@ pub fn inspect_outline(project: &Path, file: &str) -> Result<Value> {
|
||||
pub fn inspect_file(project: &Path, file: &str, lines: Option<&str>) -> Result<Value> {
|
||||
let conn = open_read(project)?;
|
||||
let fid = file_id(&conn, file)?;
|
||||
let frow = store::file_by_id(&conn, fid)?.unwrap();
|
||||
let (start, end) = parse_range(lines);
|
||||
let all = read_all_lines(project, &frow.path)?;
|
||||
let total = all.len() as i64;
|
||||
let (s, e) = (start.unwrap_or(1), end.unwrap_or(total));
|
||||
let slice = all
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(i, _)| {
|
||||
let n = (*i as i64) + 1;
|
||||
n >= s && n <= e
|
||||
})
|
||||
.map(|(_, l)| l.clone())
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
let frow =
|
||||
store::file_by_id(&conn, fid)?.with_context(|| format!("file not indexed: {file}"))?;
|
||||
let range = parse_range(lines)?;
|
||||
let (s, e) = range.unwrap_or((1, None));
|
||||
// Stream the file once: count total lines, keep only the requested slice.
|
||||
let (total, content) = read_lines_slice(project, &frow.path, s, e.unwrap_or(i64::MAX))?;
|
||||
let line_end = e.unwrap_or(total).min(total);
|
||||
Ok(serde_json::json!({
|
||||
"file": frow.path,
|
||||
"language": frow.language,
|
||||
"line_start": s,
|
||||
"line_end": e.min(total),
|
||||
"line_end": line_end,
|
||||
"total_lines": total,
|
||||
"content": slice,
|
||||
"content": content,
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -143,11 +242,131 @@ pub fn find_dead(project: &Path) -> Result<Value> {
|
||||
Ok(serde_json::json!({"count": out.len(), "files": out}))
|
||||
}
|
||||
|
||||
/// Symbols whose name never appears outside their own definition line.
|
||||
///
|
||||
/// A symbol is "dead" if, excluding import/declaration lines and its own
|
||||
/// definition, no indexed file contains its name as a whole word. Names are
|
||||
/// matched case-insensitively; same-name symbols are all kept live if any one
|
||||
/// reference exists, so this errs toward under-reporting rather than false
|
||||
/// positives. Includes `kind` so callers can filter out structural kinds
|
||||
/// (fields, variants) whose names legitimately appear only via qualified paths.
|
||||
pub fn find_dead_symbols(project: &Path) -> Result<Value> {
|
||||
let conn = open_read(project)?;
|
||||
let symbols = store::all_symbols(&conn)?;
|
||||
let files = store::all_files(&conn)?;
|
||||
let path_of: HashMap<i64, &str> = files.iter().map(|f| (f.id, f.path.as_str())).collect();
|
||||
|
||||
let mut by_name: BTreeMap<String, Vec<&store::SymbolRow>> = BTreeMap::new();
|
||||
for s in &symbols {
|
||||
by_name.entry(s.name.to_lowercase()).or_default().push(s);
|
||||
}
|
||||
|
||||
let mut line_cache: HashMap<String, Vec<String>> = HashMap::new();
|
||||
let mut dead: Vec<Value> = Vec::new();
|
||||
|
||||
for (name_lc, syms) in &by_name {
|
||||
// Reference sites for this name across every file that FTS says
|
||||
// contains it (a superset), narrowed by whole-word match.
|
||||
let mut sites: Vec<(String, i64)> = Vec::new();
|
||||
for rel in search::files_containing(&conn, name_lc, 100_000)? {
|
||||
let lines = match line_cache.entry(rel.clone()) {
|
||||
std::collections::hash_map::Entry::Occupied(e) => e.get().clone(),
|
||||
std::collections::hash_map::Entry::Vacant(e) => e
|
||||
.insert(read_all_lines(project, &rel).unwrap_or_default())
|
||||
.clone(),
|
||||
};
|
||||
for (i, line) in lines.iter().enumerate() {
|
||||
if word_matches_ci(line, name_lc) && !is_import_line(line) {
|
||||
sites.push((rel.clone(), (i as i64) + 1));
|
||||
}
|
||||
}
|
||||
}
|
||||
for s in syms {
|
||||
let own = path_of.get(&s.file_id).copied().unwrap_or("");
|
||||
let referenced = sites
|
||||
.iter()
|
||||
.any(|(rel, line)| rel != own || *line != s.line_start);
|
||||
if !referenced && !is_wired_entry_point(project, own, s) {
|
||||
dead.push(serde_json::json!({
|
||||
"id": s.id,
|
||||
"name": s.name,
|
||||
"qualified_name": s.qualified_name,
|
||||
"kind": s.kind,
|
||||
"file": own,
|
||||
"line": s.line_start,
|
||||
}));
|
||||
}
|
||||
}
|
||||
}
|
||||
dead.sort_by(|a, b| a["kind"].as_str().cmp(&b["kind"].as_str()));
|
||||
Ok(serde_json::json!({"count": dead.len(), "symbols": dead}))
|
||||
}
|
||||
|
||||
/// Whole-word, case-insensitive match (Python/Rust names may vary in case).
|
||||
fn word_matches_ci(line: &str, word: &str) -> bool {
|
||||
line.split(|c: char| !c.is_alphanumeric() && c != '_')
|
||||
.any(|tok| tok.eq_ignore_ascii_case(word))
|
||||
}
|
||||
|
||||
/// Decorator keywords that wire a symbol to an external framework. Such symbols
|
||||
/// are referenced by convention/framework, not by a name call-site, so a pure
|
||||
/// name-scan wrongly flags them dead.
|
||||
const WIRED_DECORATORS: &[&str] = &[
|
||||
"pytest",
|
||||
"fixture",
|
||||
"route",
|
||||
"router.",
|
||||
"@app.",
|
||||
"@bot.",
|
||||
"@commands.",
|
||||
"@cog.",
|
||||
"celery",
|
||||
"click.",
|
||||
"typer",
|
||||
"shared_task",
|
||||
"receiver",
|
||||
"signal",
|
||||
"listener",
|
||||
];
|
||||
|
||||
/// A symbol is externally wired if it is a pytest `test_*` function or its
|
||||
/// definition is preceded by a framework decorator (Flask/FastAPI route,
|
||||
/// pytest fixture, Celery task, click/typer command, discord.py command, …).
|
||||
fn is_wired_entry_point(project: &Path, own: &str, s: &store::SymbolRow) -> bool {
|
||||
if matches!(s.kind.as_str(), "function" | "method") && s.name.starts_with("test_") {
|
||||
return true;
|
||||
}
|
||||
let Ok(lines) = read_all_lines(project, own) else {
|
||||
return false;
|
||||
};
|
||||
let start = (s.line_start as usize).saturating_sub(1);
|
||||
let lo = start.saturating_sub(8);
|
||||
for line in lines[lo.min(lines.len())..start.min(lines.len())].iter() {
|
||||
let l = line.trim();
|
||||
let low = l.to_ascii_lowercase();
|
||||
if l.starts_with('@') && WIRED_DECORATORS.iter().any(|k| low.contains(k)) {
|
||||
return true;
|
||||
}
|
||||
// Rust attribute-wired tests: `#[test]`, `#[tokio::test]`, `#[rstest]`, …
|
||||
if l.starts_with("#[") && low.contains("test") {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// Entry-point files are legitimately never imported by anything else.
|
||||
///
|
||||
/// Only treat `main`/`lib` stems as entry points for languages where that is
|
||||
/// an actual convention — previously a `lib.py`, `lib.c` or `main.ts` would be
|
||||
/// silently hidden from `find dead`. `lib` is a Rust-only convention; `main` is
|
||||
/// meaningful for Rust, C/C++ and Python.
|
||||
fn is_entry_point(path: &str, language: &str) -> bool {
|
||||
let name = path.rsplit('/').next().unwrap_or(path);
|
||||
let stem = name.split('.').next().unwrap_or(name);
|
||||
if matches!(stem, "main" | "lib") {
|
||||
let is_main = stem == "main" && matches!(language, "rust" | "c" | "cpp" | "python");
|
||||
let is_lib = stem == "lib" && language == "rust";
|
||||
if is_main || is_lib {
|
||||
return true;
|
||||
}
|
||||
// Cargo compiles every file in these dirs as its own binary/crate.
|
||||
@@ -160,6 +379,225 @@ fn is_entry_point(path: &str, language: &str) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
/// Symbols that reference the given symbol: own file plus the import blast
|
||||
/// radius, narrowed by whole-word occurrence of the symbol name, attributed to
|
||||
/// the innermost enclosing symbol. `target` is a numeric id or an exact
|
||||
/// (case-insensitive) name; a name matches all symbols that share it.
|
||||
pub fn find_callers(project: &Path, target: &str) -> Result<Value> {
|
||||
let conn = open_read(project)?;
|
||||
let syms = resolve_symbols(&conn, target)?;
|
||||
let name = syms[0].name.clone();
|
||||
let callers = callers_of(project, &conn, &syms)?;
|
||||
let mut resp = serde_json::json!({
|
||||
"symbol": name,
|
||||
"count": callers.len(),
|
||||
"callers": callers,
|
||||
});
|
||||
if target.trim().parse::<i64>().is_err() && syms.len() > 1 {
|
||||
resp["matched"] = serde_json::json!(syms.len());
|
||||
}
|
||||
Ok(resp)
|
||||
}
|
||||
|
||||
/// Core caller scan over a set of symbols (same-name overloads share work).
|
||||
fn callers_of(project: &Path, conn: &Connection, syms: &[store::SymbolRow]) -> Result<Vec<Value>> {
|
||||
let files = store::all_files(conn)?;
|
||||
let imports = store::all_imports(conn)?;
|
||||
let graph = Graph::from_imports(&imports);
|
||||
|
||||
let mut seen: HashSet<(String, Option<i64>)> = HashSet::new();
|
||||
let mut callers: Vec<Value> = Vec::new();
|
||||
for sym in syms {
|
||||
// Candidates: the symbol's own file (intra-file calls) plus files that
|
||||
// (transitively) import it.
|
||||
let mut candidates = vec![sym.file_id];
|
||||
candidates.extend(
|
||||
graph
|
||||
.blast_radius(sym.file_id, 3)
|
||||
.into_iter()
|
||||
.map(|(id, _)| id),
|
||||
);
|
||||
for fid in candidates {
|
||||
let Some(f) = files.iter().find(|f| f.id == fid) else {
|
||||
continue;
|
||||
};
|
||||
let Ok(lines) = read_all_lines(project, &f.path) else {
|
||||
continue;
|
||||
};
|
||||
let file_syms = store::symbols_for_file(conn, fid)?;
|
||||
for (i, line) in lines.iter().enumerate() {
|
||||
if !search::word_matches(line, &sym.name) {
|
||||
continue;
|
||||
}
|
||||
// Imports/declarations reference the symbol but are not call sites.
|
||||
if is_import_line(line) {
|
||||
continue;
|
||||
}
|
||||
let lineno = (i as i64) + 1;
|
||||
let enclosing = enclosing_symbol(&file_syms, lineno);
|
||||
// Skip the symbol's own definition (and any self-recursion).
|
||||
if enclosing.map(|s| s.id) == Some(sym.id) {
|
||||
continue;
|
||||
}
|
||||
if !seen.insert((f.path.clone(), enclosing.map(|s| s.id))) {
|
||||
continue;
|
||||
}
|
||||
callers.push(serde_json::json!({
|
||||
"file": f.path,
|
||||
"line": lineno,
|
||||
"name": enclosing.map(|s| s.name.clone()),
|
||||
"kind": enclosing.map(|s| s.kind.clone()),
|
||||
"id": enclosing.map(|s| s.id),
|
||||
}));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(callers)
|
||||
}
|
||||
|
||||
/// Every line where `identifier` occurs as a whole word, across indexed files.
|
||||
/// FTS narrows candidates (a superset — tokenization may over-include); the
|
||||
/// per-line word match then yields exact line numbers.
|
||||
pub fn find_references(project: &Path, identifier: &str) -> Result<Value> {
|
||||
let conn = open_read(project)?;
|
||||
let id = identifier.trim();
|
||||
if id.is_empty() {
|
||||
return Ok(serde_json::json!({"count": 0, "references": []}));
|
||||
}
|
||||
// Definition sites (file, line) for any symbol sharing this name — these are
|
||||
// declarations, not references, and would otherwise show up as noise.
|
||||
let mut defs: HashSet<(String, i64)> = HashSet::new();
|
||||
for s in store::symbols_by_name(&conn, id)? {
|
||||
if let Some(f) = store::file_by_id(&conn, s.file_id)? {
|
||||
defs.insert((f.path, s.line_start));
|
||||
}
|
||||
}
|
||||
let mut out: Vec<Value> = Vec::new();
|
||||
for rel in search::files_containing(&conn, id, 100_000)? {
|
||||
let Ok(lines) = read_all_lines(project, &rel) else {
|
||||
continue;
|
||||
};
|
||||
for (i, line) in lines.iter().enumerate() {
|
||||
let line_no = (i as i64) + 1;
|
||||
if defs.contains(&(rel.clone(), line_no)) {
|
||||
continue;
|
||||
}
|
||||
let code = code_only_line(line);
|
||||
if search::word_matches(&code, id) {
|
||||
out.push(serde_json::json!({
|
||||
"file": rel,
|
||||
"line": line_no,
|
||||
"text": line.trim().to_string(),
|
||||
}));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(serde_json::json!({"count": out.len(), "references": out}))
|
||||
}
|
||||
|
||||
/// Line with string literals and comments blanked so a subsequent whole-word
|
||||
/// match only hits real code references. Multi-line docstrings/block comments
|
||||
/// are handled line-by-line as a heuristic.
|
||||
fn code_only_line(line: &str) -> String {
|
||||
let trimmed = line.trim_start();
|
||||
if trimmed.starts_with("//")
|
||||
|| trimmed.starts_with('#')
|
||||
|| trimmed.starts_with("--")
|
||||
|| trimmed.starts_with("/*")
|
||||
|| trimmed.starts_with("'''")
|
||||
|| trimmed.starts_with("\"\"\"")
|
||||
{
|
||||
return String::new();
|
||||
}
|
||||
let bytes = line.as_bytes();
|
||||
let mut out = String::with_capacity(line.len());
|
||||
let mut i = 0;
|
||||
while i < bytes.len() {
|
||||
let c = bytes[i] as char;
|
||||
if c == '/' && bytes.get(i + 1) == Some(&b'/') {
|
||||
break; // line comment
|
||||
}
|
||||
if c == '/' && bytes.get(i + 1) == Some(&b'*') {
|
||||
match line[i + 2..].find("*/") {
|
||||
Some(end) => {
|
||||
i += 2 + end + 2;
|
||||
out.push(' ');
|
||||
continue;
|
||||
}
|
||||
None => break,
|
||||
}
|
||||
}
|
||||
if c == '"' || c == '\'' || c == '`' {
|
||||
let ch = c;
|
||||
// Rust raw string `r###"..."###`: the `#` run before the quote is part
|
||||
// of the delimiter, so the closing marker is `"###`.
|
||||
let mut hashes = 0usize;
|
||||
let mut j = i;
|
||||
while j > 0 && bytes[j - 1] == b'#' {
|
||||
hashes += 1;
|
||||
j -= 1;
|
||||
}
|
||||
if ch == '"' && hashes > 0 {
|
||||
let delim = format!("\"{}", "#".repeat(hashes));
|
||||
match line[i + 1..].find(&delim) {
|
||||
Some(end) => {
|
||||
i += 1 + end + delim.len();
|
||||
out.push(' ');
|
||||
continue;
|
||||
}
|
||||
None => break,
|
||||
}
|
||||
}
|
||||
let triple = line[i..].starts_with(&format!("{ch}{ch}{ch}"));
|
||||
let close = if triple {
|
||||
line[i + 3..]
|
||||
.find(&format!("{ch}{ch}{ch}"))
|
||||
.map(|p| i + 3 + p + 3)
|
||||
} else {
|
||||
line[i + 1..].find(ch).map(|p| i + 1 + p + 1)
|
||||
};
|
||||
match close {
|
||||
Some(end) => {
|
||||
i = end;
|
||||
out.push(' ');
|
||||
continue;
|
||||
}
|
||||
None => break, // unterminated string: rest is string
|
||||
}
|
||||
}
|
||||
out.push(c);
|
||||
i += 1;
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// True if `line` is an import/declaration (a reference, not a call site).
|
||||
fn is_import_line(line: &str) -> bool {
|
||||
let l = line.trim_start();
|
||||
[
|
||||
"use ",
|
||||
"pub use ",
|
||||
"pub(crate) use ",
|
||||
"pub(super) use ",
|
||||
"extern crate ",
|
||||
"import ",
|
||||
"from ",
|
||||
"#include",
|
||||
"using ",
|
||||
"mod ",
|
||||
"type ",
|
||||
]
|
||||
.iter()
|
||||
.any(|p| l.starts_with(p))
|
||||
}
|
||||
|
||||
/// Innermost symbol (largest line_start) whose span contains `line`.
|
||||
fn enclosing_symbol(syms: &[store::SymbolRow], line: i64) -> Option<&store::SymbolRow> {
|
||||
syms.iter()
|
||||
.filter(|s| s.line_start <= line && line <= s.line_end)
|
||||
.max_by_key(|s| s.line_start)
|
||||
}
|
||||
|
||||
pub fn inspect_cycles(project: &Path) -> Result<Value> {
|
||||
let conn = open_read(project)?;
|
||||
let files = store::all_files(&conn)?;
|
||||
@@ -232,17 +670,7 @@ fn file_id(conn: &Connection, path: &str) -> Result<i64> {
|
||||
}
|
||||
|
||||
fn read_lines(project: &Path, rel: &str, start: i64, end: i64) -> Result<String> {
|
||||
let all = read_all_lines(project, rel)?;
|
||||
Ok(all
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(i, _)| {
|
||||
let n = (*i as i64) + 1;
|
||||
n >= start && n <= end
|
||||
})
|
||||
.map(|(_, l)| l.clone())
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n"))
|
||||
Ok(read_lines_slice(project, rel, start, end)?.1)
|
||||
}
|
||||
|
||||
fn read_all_lines(project: &Path, rel: &str) -> Result<Vec<String>> {
|
||||
@@ -252,13 +680,71 @@ fn read_all_lines(project: &Path, rel: &str) -> Result<Vec<String>> {
|
||||
Ok(content.lines().map(|l| l.to_string()).collect())
|
||||
}
|
||||
|
||||
fn parse_range(lines: Option<&str>) -> (Option<i64>, Option<i64>) {
|
||||
let Some(s) = lines else { return (None, None) };
|
||||
/// Stream `rel`, returning the total line count and lines `[start, end]`
|
||||
/// (1-based, inclusive) joined. Reads the file once without materializing every
|
||||
/// line.
|
||||
fn read_lines_slice(project: &Path, rel: &str, start: i64, end: i64) -> Result<(i64, String)> {
|
||||
let path = project.join(rel);
|
||||
let file =
|
||||
std::fs::File::open(&path).with_context(|| format!("cannot read {}", path.display()))?;
|
||||
let reader = std::io::BufReader::new(file);
|
||||
let mut collected: Vec<String> = Vec::new();
|
||||
let mut total = 0i64;
|
||||
for line in reader.lines() {
|
||||
let line = line.with_context(|| format!("cannot read {}", path.display()))?;
|
||||
total += 1;
|
||||
if total >= start && total <= end {
|
||||
collected.push(line);
|
||||
}
|
||||
}
|
||||
Ok((total, collected.join("\n")))
|
||||
}
|
||||
|
||||
/// Parse and validate a `--lines` range. `None` → whole file. Accepts `a`
|
||||
/// (line a to end) or `a-b`. Rejects non-numeric or reversed ranges.
|
||||
fn parse_range(lines: Option<&str>) -> Result<Option<(i64, Option<i64>)>> {
|
||||
let Some(s) = lines else { return Ok(None) };
|
||||
let s = s.trim();
|
||||
if s.is_empty() {
|
||||
bail!("empty line range");
|
||||
}
|
||||
let parts: Vec<&str> = s.split('-').collect();
|
||||
let start = parts.first().and_then(|p| p.trim().parse::<i64>().ok());
|
||||
let end = parts.get(1).and_then(|p| p.trim().parse::<i64>().ok());
|
||||
let (start, end) = match parts.as_slice() {
|
||||
[a] => {
|
||||
let start = a
|
||||
.trim()
|
||||
.parse::<i64>()
|
||||
.with_context(|| format!("invalid line: {a}"))?;
|
||||
(start, None)
|
||||
}
|
||||
[a, b] => {
|
||||
let start = a
|
||||
.trim()
|
||||
.parse::<i64>()
|
||||
.with_context(|| format!("invalid range start: {a}"))?;
|
||||
let end = if b.trim().is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(
|
||||
b.trim()
|
||||
.parse::<i64>()
|
||||
.with_context(|| format!("invalid range end: {b}"))?,
|
||||
)
|
||||
};
|
||||
(start, end)
|
||||
}
|
||||
_ => bail!("invalid line range: {s} (expected 'start-end')"),
|
||||
};
|
||||
if start < 1 {
|
||||
bail!("line range start must be >= 1: {s}");
|
||||
}
|
||||
if let Some(end) = end
|
||||
&& end < start
|
||||
{
|
||||
bail!("line range end must be >= start: {s}");
|
||||
}
|
||||
Ok(Some((start, end)))
|
||||
}
|
||||
|
||||
// ---- tree ----
|
||||
|
||||
@@ -339,3 +825,28 @@ fn raw_to_nodes(node: &RawNode, base: &str, depth: Option<usize>) -> Vec<TreeNod
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::is_entry_point;
|
||||
|
||||
#[test]
|
||||
fn entry_point_is_language_scoped() {
|
||||
// Rust: main.rs and lib.rs are entry points.
|
||||
assert!(is_entry_point("src/main.rs", "rust"));
|
||||
assert!(is_entry_point("src/lib.rs", "rust"));
|
||||
// C/C++ and Python: `main` is a convention, `lib` is not.
|
||||
assert!(is_entry_point("main.c", "c"));
|
||||
assert!(is_entry_point("main.cpp", "cpp"));
|
||||
assert!(is_entry_point("main.py", "python"));
|
||||
// `lib` outside Rust is just a module/file named lib — reportable dead.
|
||||
assert!(!is_entry_point("lib.py", "python"));
|
||||
assert!(!is_entry_point("lib.c", "c"));
|
||||
// `main` is not a convention for JS/TS — reportable dead.
|
||||
assert!(!is_entry_point("main.ts", "typescript"));
|
||||
assert!(!is_entry_point("lib.ts", "typescript"));
|
||||
// Rust integration-test / example files are entry points regardless of name.
|
||||
assert!(is_entry_point("tests/integration.rs", "rust"));
|
||||
assert!(is_entry_point("src/bin/tool.rs", "rust"));
|
||||
}
|
||||
}
|
||||
+82
-26
@@ -23,13 +23,14 @@ pub struct SymbolHit {
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct TextHit {
|
||||
pub file: String,
|
||||
pub line: i64,
|
||||
pub snippet: String,
|
||||
}
|
||||
|
||||
const SYMBOL_COLS: &str =
|
||||
"s.id, s.name, s.qualified_name, s.kind, s.signature, f.path, s.line_start, s.line_end";
|
||||
|
||||
pub fn search_symbols(conn: &Connection, query: &str) -> Result<Vec<SymbolHit>> {
|
||||
pub fn search_symbols(conn: &Connection, query: &str, limit: usize) -> Result<Vec<SymbolHit>> {
|
||||
let q = query.trim();
|
||||
let mut seen: HashSet<i64> = HashSet::new();
|
||||
let mut out: Vec<SymbolHit> = Vec::new();
|
||||
@@ -41,10 +42,10 @@ pub fn search_symbols(conn: &Connection, query: &str) -> Result<Vec<SymbolHit>>
|
||||
{
|
||||
let mut stmt = conn.prepare(&format!(
|
||||
"SELECT {SYMBOL_COLS} FROM symbols s JOIN files f ON f.id = s.file_id \
|
||||
WHERE s.name = ?1 COLLATE NOCASE ORDER BY s.id LIMIT 200"
|
||||
WHERE s.name = ?1 COLLATE NOCASE ORDER BY s.id LIMIT ?2"
|
||||
))?;
|
||||
let rows = stmt.query_map(params![q], |r| row_to_hit(r, "exact"))?;
|
||||
push_hits(&mut out, &mut seen, rows)?;
|
||||
let rows = stmt.query_map(params![q, limit as i64], |r| row_to_hit(r, "exact"))?;
|
||||
push_hits(&mut out, &mut seen, rows, limit)?;
|
||||
}
|
||||
|
||||
// 2) prefix
|
||||
@@ -52,10 +53,10 @@ pub fn search_symbols(conn: &Connection, query: &str) -> Result<Vec<SymbolHit>>
|
||||
let pat = format!("{}%", q);
|
||||
let mut stmt = conn.prepare(&format!(
|
||||
"SELECT {SYMBOL_COLS} FROM symbols s JOIN files f ON f.id = s.file_id \
|
||||
WHERE s.name LIKE ?1 COLLATE NOCASE ORDER BY s.id LIMIT 200"
|
||||
WHERE s.name LIKE ?1 COLLATE NOCASE ORDER BY s.id LIMIT ?2"
|
||||
))?;
|
||||
let rows = stmt.query_map(params![pat], |r| row_to_hit(r, "prefix"))?;
|
||||
push_hits(&mut out, &mut seen, rows)?;
|
||||
let rows = stmt.query_map(params![pat, limit as i64], |r| row_to_hit(r, "prefix"))?;
|
||||
push_hits(&mut out, &mut seen, rows, limit)?;
|
||||
}
|
||||
|
||||
// 3) fulltext over name/qualified_name/kind/signature
|
||||
@@ -65,32 +66,49 @@ pub fn search_symbols(conn: &Connection, query: &str) -> Result<Vec<SymbolHit>>
|
||||
"SELECT {SYMBOL_COLS} FROM symbols_fts \
|
||||
JOIN symbols s ON s.id = symbols_fts.rowid \
|
||||
JOIN files f ON f.id = s.file_id \
|
||||
WHERE symbols_fts MATCH ?1 ORDER BY rank LIMIT 200"
|
||||
WHERE symbols_fts MATCH ?1 ORDER BY rank LIMIT ?2"
|
||||
))?;
|
||||
let rows = stmt.query_map(params![phrase], |r| row_to_hit(r, "fulltext"))?;
|
||||
push_hits(&mut out, &mut seen, rows)?;
|
||||
let rows = stmt.query_map(params![phrase, limit as i64], |r| row_to_hit(r, "fulltext"))?;
|
||||
push_hits(&mut out, &mut seen, rows, limit)?;
|
||||
}
|
||||
|
||||
// 4) substring fallback — catches typos and mid-word queries
|
||||
{
|
||||
let pat = format!("%{}%", q);
|
||||
let mut stmt = conn.prepare(&format!(
|
||||
"SELECT {SYMBOL_COLS} FROM symbols s JOIN files f ON f.id = s.file_id \
|
||||
WHERE s.name LIKE ?1 COLLATE NOCASE ORDER BY s.id LIMIT ?2"
|
||||
))?;
|
||||
let rows = stmt.query_map(params![pat, limit as i64], |r| row_to_hit(r, "substring"))?;
|
||||
push_hits(&mut out, &mut seen, rows, limit)?;
|
||||
}
|
||||
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
pub fn search_text(conn: &Connection, query: &str) -> Result<Vec<TextHit>> {
|
||||
let q = query.trim();
|
||||
if q.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
/// Report `(line_number, trimmed_line)` for every line containing the query.
|
||||
/// A line matches if it contains the whole query (case-insensitive) or, for a
|
||||
/// multi-word query, all of its terms.
|
||||
pub fn lines_matching(content: &str, q: &str) -> Vec<(i64, String)> {
|
||||
let needle = q.to_lowercase();
|
||||
let terms: Vec<&str> = needle.split_whitespace().collect();
|
||||
let mut out = Vec::new();
|
||||
for (i, line) in content.lines().enumerate() {
|
||||
let l = line.to_lowercase();
|
||||
if l.contains(&needle) || terms.iter().all(|t| l.contains(t)) {
|
||||
out.push(((i + 1) as i64, clip_line(line.trim())));
|
||||
}
|
||||
let phrase = fts_phrase(q);
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT path, snippet(content_fts, 1, '«', '»', '…', 24) \
|
||||
FROM content_fts WHERE content_fts MATCH ?1 ORDER BY rank LIMIT 100",
|
||||
)?;
|
||||
let rows = stmt.query_map(params![phrase], |r| {
|
||||
Ok(TextHit {
|
||||
file: r.get(0)?,
|
||||
snippet: r.get(1)?,
|
||||
})
|
||||
})?;
|
||||
Ok(rows.collect::<std::result::Result<Vec<_>, _>>()?)
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Trim a snippet line to a readable length.
|
||||
fn clip_line(s: &str) -> String {
|
||||
let mut out: String = s.chars().take(200).collect();
|
||||
if s.chars().count() > 200 {
|
||||
out.push('…');
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn row_to_hit(
|
||||
@@ -114,8 +132,12 @@ fn push_hits(
|
||||
out: &mut Vec<SymbolHit>,
|
||||
seen: &mut HashSet<i64>,
|
||||
rows: impl Iterator<Item = std::result::Result<SymbolHit, rusqlite::Error>>,
|
||||
max: usize,
|
||||
) -> Result<()> {
|
||||
for r in rows {
|
||||
if out.len() >= max {
|
||||
break;
|
||||
}
|
||||
let hit = r?;
|
||||
if seen.insert(hit.id) {
|
||||
out.push(hit);
|
||||
@@ -128,3 +150,37 @@ fn push_hits(
|
||||
fn fts_phrase(q: &str) -> String {
|
||||
format!("\"{}\"", q.replace('"', "\"\""))
|
||||
}
|
||||
|
||||
/// Whole-word (identifier) match. `_` is treated as part of an identifier, so
|
||||
/// `authenticate` matches `authenticate` and `auth::authenticate`, but not
|
||||
/// `authenticate_user`.
|
||||
pub fn word_matches(line: &str, word: &str) -> bool {
|
||||
line.split(|c: char| !c.is_alphanumeric() && c != '_')
|
||||
.any(|tok| tok == word)
|
||||
}
|
||||
|
||||
/// Relative paths of files whose content contains the phrase (FTS superset).
|
||||
/// Tokenization may over-include (e.g. `authenticate_user` contains an
|
||||
/// `authenticate` token); callers refine with [`word_matches`] for exactness.
|
||||
pub fn files_containing(conn: &Connection, query: &str, limit: usize) -> Result<Vec<String>> {
|
||||
let phrase = fts_phrase(query);
|
||||
let mut stmt =
|
||||
conn.prepare("SELECT DISTINCT path FROM content_fts WHERE content_fts MATCH ?1 LIMIT ?2")?;
|
||||
let rows = stmt.query_map(params![phrase, limit as i64], |r| r.get(0))?;
|
||||
Ok(rows.collect::<std::result::Result<Vec<_>, _>>()?)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::word_matches;
|
||||
|
||||
#[test]
|
||||
fn word_matches_is_identifier_aware() {
|
||||
assert!(word_matches("auth::authenticate(token)", "authenticate"));
|
||||
assert!(word_matches("self.authenticate()", "authenticate"));
|
||||
assert!(word_matches("let authenticate = 1;", "authenticate"));
|
||||
assert!(!word_matches("authenticate_user()", "authenticate"));
|
||||
assert!(!word_matches("reauthenticate()", "authenticate"));
|
||||
assert!(!word_matches("", "authenticate"));
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user