Skip to content

python: consolidated gaps from agent dogfooding (test selection, framework reach) #1832

Open

Description

@swapnilpaliwal-sd

Dogfooding impact, path and test-impact on six Python projects (a Flask app, a FastAPI app, a REST framework, a packaging CLI, a crawling framework, an HTTP client library), each answer checked against grep or a raise probe. Open items only; route to test client came up in 3 of 4 rounds.

Root cause

  • A handler reached through a string (HTTP route, CLI subcommand, template name, fixture name) has no edge from the code that uses the string, so a test driving it through a test client or a command line is not selected.
  • The test walk is too wide: class and module bodies count as call nodes, and a same-name call on a receiver of a known different type still seeds the transitive layer.
  • A function or bound method passed as a value is neither a reference nor a path hop, and a decorator wrapper's f(*args) binds to every function it wraps.
bp = Blueprint("items", __name__, url_prefix="/api/items")

@bp.post("/<int:item_id>/tags")
def add_tag(item_id): ...

def test_add_tag(client):
    client.post(f"/api/items/{7}/tags", json={"tag": "x"})

A handler reached only this way gets 0 tests from impact --tests, and path <test> <handler> says no chain.

In progress: fix/python-dogfood-r1 (dispatch hop by dotted name, nested defs, registry returns), fix/framework-gaps-0929 (request.getfixturevalue()).

Checklist

  • Flask, Flask-RESTX, Django and FastAPI @router routes (with prefix) match URL literals and f-string helpers in client.get/post(...). Flask: path <test file> <Resource>.post says no chain; FastAPI: impact <handler> gives 0 tests, grep finds 2.
  • Class and module bodies are not call nodes. HTTP client library: a regex edit in one lexer class selects 427 of 429 tests via TestX.<classbody> -> SomeLexer.<classbody>.
  • A same-name call on a known different receiver type seeds nothing. Packaging CLI: impact <Cache>.cleanup pulls in tempdir.cleanup() and a closure, about 146 wrong tests.
  • A decorator wrapper's f(*args) does not bind to every decorated function. Flask: 24 of 26 chains for one model method run through an auth wrapper into an unrelated post. Re-check after fix/python-dogfood-r1.
  • cache.delete_memoized(fn) is a reference, not a call; @cache.memoize does not hide real calls; a bare fn() is not "untyped receiver".
  • @property reads on a typed receiver are calls; override rows come from class bases (build prints "0 override rows"). REST framework: path <test> <Field>.<getter> says no chain.
  • {{ x|f }} reaches the view whose template_name names that template, then its tests; filter-registered functions resolve.
  • argparse and click registration (add_parser, register_to(subparsers), click groups) links a test's command line to the handler. Packaging CLI: impact <Command>.<method> --tests gives 0 of 891; a raise probe fails one.
  • signals.connect(o.handler, ...) is a rename site and a path hop; settings-enabled extensions reach their receivers. Crawling framework: path <Ext>.<factory> <Ext>.<handler> says no chain.
  • request.fixturenames, usefixtures("x") strings and from m import f lines are rename sites.
  • test-impact --files a b does not take --files as a file; with page=all or limit it prints every selected id (today 6 of 107) and calls a parameter reorder a reorder.

Related, distinct and left open: #1766 (a script started by a common-word file name).

Activity

  1. swapnilpaliwal-sd commented on Sep 30, 2026

    @swapnilpaliwal-sd
    ContributorAuthor

    Folded in from #1766:

    • test-impact <script>.py lists every test that starts the script by file name, also when the stem is a common word. Repro: test-impact plugins/axiomcode/hooks/direct.py lists none of the 9 test files that spell 'direct.py' and 2 unrelated ones that only contain the word; enrich.py in the same directory gets all its test files.
  2. swapnilpaliwal-sd commented on Oct 15, 2026

    @swapnilpaliwal-sd
    ContributorAuthor

    Sub-issues filed for gaps in this list, each closed by its own pull request (this issue stays open):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpythonPythonskillAgent-facing query skill (skills/axiomcode)

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions