feat: scope evaluators to the current project on create

[quinn-galileo]

Why

With metric RBAC on (now the server default), a create request that carries neither is_global nor project_ids is treated as global, and only org admins may create global metrics. The SDK never sent scope, so every non-admin evaluator create returned 403. This PR adds scope to every SDK evaluator create path and defaults it to the user's current project. Ticket: SAO-9342.

What changed

New project_ids: list[str] | None and is_global: bool | None on Evaluators.create_custom_llm_evaluator, Evaluators.create_decision_evaluator, the module-level create_custom_llm_evaluator / create_decision_evaluator (trailing keyword params, positional order unchanged), and keyword-only on LlmEvaluator, CodeEvaluator and DecisionEvaluator. Evaluator.is_global and Evaluator.scope_projects report only what the API returned: None / [] until the evaluator is populated by create(), get(), list() or refresh(). The scope you pass to a constructor is kept privately and read only by create(), so calling create() on an evaluator fetched with get() resolves scope from the current project, not from the fetched server values.

The scope sent on create, in order:

Arguments / environment	Request
is_global=True and non-empty project_ids	ValidationError, nothing sent
is_global=True	is_global=True, no project_ids
project_ids=[...]	is_global=False, those ids (deduplicated, order kept)
project_ids=[]	ValidationError
neither, a current project is set (decorator context project, then SPLUNK_AO_PROJECT_ID, then SPLUNK_AO_PROJECT)	is_global=False, project_ids=[<that project's id>]
current project set but not found	NotFoundError, nothing sent (no silent fallback to global)
is_global=False, no project_ids, no current project	ValidationError
nothing at all	no scope sent; the server default (global, org admins only) applies

The current project must already exist. Unlike logging, creating an evaluator does not create it, and the NotFoundError says so and points at project_ids=[...] or is_global=True.

Behaviour change for admins: an admin with a current project set now gets a project-scoped evaluator. Pass is_global=True for a global one. The CHANGELOG says so, and names 0.6.0 as the minimum splunk-ao version for creating project-scoped evaluators.

Where to start reading

• _resolve_create_scope in src/splunk_ao/evaluators.py holds all the rules above.
• src/splunk_ao/evaluator.py: the constructors and create() of LlmEvaluator, CodeEvaluator and DecisionEvaluator, and _populate_from_scorer_response.
• Scope is resolved before code validation and before any write.

Testing

poetry run pytest tests/test_evaluators.py tests/test_evaluator.py tests/test_decision_scorers.py -n 0 (254 passed, including tests/test_projects.py) and poetry run invoke type-check (clean). All network is mocked. Tests cover the default, each precedence step, explicit override, global, nothing-set, every validation error, not-found, constructor forwarding, and population from the response, for the LLM, code and decision paths.

Related PRs

Repo	PR	Purpose	Status
splunk/splunk-ao-python	#279	regenerate the API client from the production spec (this PR's base)	in review
splunk/splunk-ao-python	#274 (this PR)	scope evaluators to the current project on create	draft
splunk/splunk-ao-python	#275	clearer permission errors and a scope-change helper, stacked on this PR	draft

🤖 Generated with Claude Code

[github-actions]

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}

[quinn-galileo]

🤖 This review was generated by the Astra agent (claude-opus-5-5). It may contain mistakes.

Verdict: approve — Commit 49bef14 resolved the CHANGELOG conflict cleanly. The scope resolution is correct and tested on every create path, and I found no new defects.

Addressed before this review

• Merge commit 96f1114 committed conflict markers in CHANGELOG.md, and a naive resolution would have put the decision-evaluator entries under Changed. Fixed in 49bef14. No markers remain anywhere in the repository. The sections are now ordered Added, Changed, Fixed, and main's entries are intact.

[quinn-galileo]

🤖 This review was generated by the Astra agent (claude-opus-5-5). It may contain mistakes.

Verdict: approve — Commit 9fcea53 now keeps the requested scope private and handles a missing is_global correctly. I traced the error rewrap and found no new defects.