endergebnis/rustytokens
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Expose or explicitly scope the missing Toktoken analysis surface #6

New Issue
Closed
opened 2026-09-04 07:52:36 +02:00 by nami-ai · 1 comment
nami-ai commented 2026-09-04 07:52:36 +02:00
Collaborator
Copy link
Copy source

Toktoken 0.5.0 exposes 27 tools; rustytokens currently advertises and implements 13 (README.md:99-102). The core analysis/ergonomics gaps are:

  • index_file
  • find_callers
  • find_references
  • inspect_bundle
  • inspect_dependencies
  • inspect_hierarchy
  • search_cooccurrence
  • search_similar

Existing operations also have narrower semantics than Toktoken: find_dead reports only file-level no-importer nodes (src/ops.rs:126-143), inspect_blast_radius accepts a file rather than a symbol and returns only file-level reverse dependencies (src/ops.rs:183-200), and stats omits staleness/token-savings data. There are no result limits/filters/detail controls on the search operations.

If rustytokens is intended to be a compatible implementation, add the core tools and align their schemas/semantics. If the smaller scope is intentional, document the compatibility boundary. The convenience/management tools (codebase_detect, help, suggest, projects_list, cache_clear, index_github) can remain out of scope unless full tool parity is required.

Toktoken 0.5.0 exposes 27 tools; rustytokens currently advertises and implements 13 (`README.md:99-102`). The core analysis/ergonomics gaps are: - `index_file` - `find_callers` - `find_references` - `inspect_bundle` - `inspect_dependencies` - `inspect_hierarchy` - `search_cooccurrence` - `search_similar` Existing operations also have narrower semantics than Toktoken: `find_dead` reports only file-level no-importer nodes (`src/ops.rs:126-143`), `inspect_blast_radius` accepts a file rather than a symbol and returns only file-level reverse dependencies (`src/ops.rs:183-200`), and `stats` omits staleness/token-savings data. There are no result limits/filters/detail controls on the search operations. If rustytokens is intended to be a compatible implementation, add the core tools and align their schemas/semantics. If the smaller scope is intentional, document the compatibility boundary. The convenience/management tools (`codebase_detect`, `help`, `suggest`, `projects_list`, `cache_clear`, `index_github`) can remain out of scope unless full tool parity is required.
endergebnis commented 2026-09-04 20:37:41 +02:00
Owner
Copy link
Copy source

Fixed in commit 4338e28.

Fixed in commit 4338e28.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
nami-ai endergebnis
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: endergebnis/rustytokens#6