You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[FEATURE] LLM_EXTRA_BODY: extra fields in the LLM request body #1494
Is your feature request related to a problem? Please describe.
Some provider parameters go in the request body, but Strix does not have a setting to set them. For example two OpenAI parameters are very relevant for running Strix:
service_tier: "flex": Flex processing is cheaper in exchange for slower responses.
safety_identifier: OpenAI runs cybersecurity checks on API traffic, and may temporarily restrict access to the affected models when they trigger. Without a per-user safety identifier, "access may be temporarily revoked for the entire organization". With one, OpenAI can restrict the affected user instead, which reduces disruption for everyone else. Because of this, my organization requires this parameter.
Sending those as headers doesn't work, OpenAI requires them to be in the body.
Describe the solution you'd like
A LLM_EXTRA_BODY setting, a JSON object merged into the body of every request to the configured model, next to LLM_EXTRA_HEADERS and with the same rules:
export LLM_EXTRA_BODY='{"service_tier":"flex","safety_identifier":"<sha256 of the user email>"}'
It applies on the LiteLLM and the native OpenAI routes.
It applies to every call to the configured model: agents, preflight, history compaction and finding deduplication.
A dedicated dedupe model gets its own DEDUPE_LLM_EXTRA_BODY.
Describe alternatives you've considered
none
Additional context
I have a working implementation with tests and will open a PR.
Activity