Skip to content

[FEATURE] LLM_EXTRA_BODY: extra fields in the LLM request body #1494

Open

Description

@SzymonNastaly

Is your feature request related to a problem? Please describe.
Some provider parameters go in the request body, but Strix does not have a setting to set them. For example two OpenAI parameters are very relevant for running Strix:

  • service_tier: "flex": Flex processing is cheaper in exchange for slower responses.
  • safety_identifier: OpenAI runs cybersecurity checks on API traffic, and may temporarily restrict access to the affected models when they trigger. Without a per-user safety identifier, "access may be temporarily revoked for the entire organization". With one, OpenAI can restrict the affected user instead, which reduces disruption for everyone else. Because of this, my organization requires this parameter.

Sending those as headers doesn't work, OpenAI requires them to be in the body.

Describe the solution you'd like
A LLM_EXTRA_BODY setting, a JSON object merged into the body of every request to the configured model, next to LLM_EXTRA_HEADERS and with the same rules:

export LLM_EXTRA_BODY='{"service_tier":"flex","safety_identifier":"<sha256 of the user email>"}'
  • It applies on the LiteLLM and the native OpenAI routes.
  • It applies to every call to the configured model: agents, preflight, history compaction and finding deduplication.
  • A dedicated dedupe model gets its own DEDUPE_LLM_EXTRA_BODY.

Describe alternatives you've considered
none

Additional context
I have a working implementation with tests and will open a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions