Skip to main content
POST
/
v1
/
orgs
/
{org_id}
/
scenarios
/
calibrate
Calibrate Judge
curl --request POST \
  --url https://api.letpiper.com/v1/orgs/{org_id}/scenarios/calibrate \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "examples": [
    {
      "id": "<string>",
      "user_message": "<string>",
      "agent_response": "<string>",
      "criterion_question": "<string>",
      "human_verdict": "PASS",
      "human_critique": "<string>",
      "tags": [
        "<string>"
      ]
    }
  ]
}
'
{
  "total": 123,
  "agreements": 123,
  "agreement_rate": 123,
  "true_positives": 123,
  "true_negatives": 123,
  "false_positives": 123,
  "false_negatives": 123,
  "tpr": 123,
  "tnr": 123,
  "results": [
    {
      "example_id": "<string>",
      "criterion": "<string>",
      "human_verdict": "PASS",
      "judge_verdict": "PASS",
      "agrees": true,
      "judge_critique": "<string>"
    }
  ]
}

Authorizations

Authorization
string
header
required

Platform token (starts with pat_)

Path Parameters

org_id
string<uuid>
required

Query Parameters

agent_id
string<uuid>
required

Body

application/json

Request schema for running calibration.

examples
LabeledExample · object[]
required

Human-labeled examples to calibrate against

Minimum array length: 1

Response

Successful Response

Aggregate calibration metrics.

total
integer
required
agreements
integer
required
agreement_rate
number
required
true_positives
integer
required
true_negatives
integer
required
false_positives
integer
required
false_negatives
integer
required
tpr
number
required

True Positive Rate (sensitivity)

tnr
number
required

True Negative Rate (specificity)

results
CalibrationResultItem · object[]
required