Report triage

Triage of a user report about a post, where rules, a model and a moderator decide together and every step is on the record: your platform sends the classifier score, any known-content match and the post's text with the report, events count the author's strikes and reports, a model reads the post and names the harm and how urgent it is, and what neither the rules nor the model clear goes to a moderator. Pack: examples/policies/report-triage/.

A pattern for expressing a trust and safety policy, not legal guidance: the thresholds, questions and fields are placeholders chosen to make the example run; a real policy comes from the business's own community rules. Its read step calls one model app install, chatgpt (The model step).

The decision it automates

Whether to leave the content up (approve), send it to a moderator, or remove it (reject), when your platform starts the run. Only a rule removes content on its own: the model's doubt sends a report to a moderator, never to removal.

Data expected

• The author as a subject (kind user) whose lastReport your platform writes before it starts the run: classifierScore and knownContentMatch for the rules, and text (the reported post), reason (the reason the reporter chose) and comment (what the reporter wrote) for the model. Without text, the model is not asked and the rules decide alone.
• report.created and strike.issued events on the author, posted through POST /events.

Steps and rules

Step	Type	Why
rules	evaluate_rules	The report-triage rule set under max_severity; nothing is collected.
route	branch	rules.outcome == reject (known content) removes at once; a report with lastReport.text goes to read; without one, manual_review opens a case and the rest stays up.
read	call_app	chatgpt's decide: three questions about the post, answered under ai (The model step).
triage	branch	rules.outcome == manual_review opens a case; ai.answers.within_rules.probability >= 0.8 leaves the post up; the default (a model that sees harm, or is unsure) opens a case.
review	create_case	moderation, priority high, or critical when the model's urgency says so (priorityFrom: ai.answers.urgency.value), 1-hour SLA, queued by the model's concern (queueFrom: ai.answers.concern.value).
decide	emit_decision	From the rules, or the moderator's decision.

The rule set report-triage (useCase: monitoring) uses max_severity: a failed block rule rejects, a failed warn rule sends the run to review, and the weights add up to the score.

Rule	Type	Severity, weight	Fails when
report_known_content	comparison	block, 100	lastReport.knownContentMatch is not false
report_classifier_score	score threshold	warn, 40	lastReport.classifierScore is above 0.8
report_repeat_offender	velocity	warn, 30	more than two strike.issued events in 90 days
report_burst	velocity	warn, 20	more than ten report.created events in 24 hours

The model step

read asks the model three questions about the post. Its state is the step's variables as one JSON object, mapped from subject.lastReport: post (its text), reason and comment. A field your platform leaves out is left out.

Question	Type	Asks	Used by
within_rules	yes or no, threshold 0.8	Does the post stay within ordinary community rules: no harassment, hate, threats, encouragement of self-harm, non-consensual sexual content or spam?	triage: 0.8 or more leaves up a post every rule passed
concern	one of harassment, hate, threat, self_harm, sexual, spam, none	Which kind of harm does the report most likely concern?	review: the case's queue
urgency	one of high, critical	How soon should a moderator look? critical means someone may be in danger	review: the case's priority

The model can send a report the rules would close to a moderator, never close one they send: known content never reaches it, a report the rules send to a moderator goes there whatever the model answers, and its yes leaves up only a post every rule passed. It can raise a case to critical, never lower one below the step's high. The questions name kinds of harm under your community rules; none asks whether a post is lawful, and no answer is a legal determination.

What the vendor sees

read sends OpenAI, through the chatgpt install, the three questions (their instructions, criteria and options) and the state: the reported post, the reason the reporter chose and what they wrote, and nothing else of the run, the author's name and the reporter's identity included. The post and the comment may name people. Both are text the model reads: an instruction written into the post at worst gets the post left up where the rules alone would have left it up (Prompt injection). OpenAI's own terms say what it keeps; the console's preset calls its Responses API with storing off, decisions included (Data protection).

The chatgpt install

The pack declares the install it calls, "installs": { "chatgpt": "chatgpt" }; installs are never part of a pack. The admin console's import result installs chatgpt in one click: ChatGPT (OpenAI API) on OpenAI's Responses API, with a sample key in place of yours. Replace the key with your OpenAI API key on the install's page (Connect › Apps › ChatGPT (OpenAI API) › chatgpt, under Keys) or through POST /app-installs/chatgpt/secrets/openaiApiKey.

Until then, a run whose path reaches read fails there and no case opens; so does a run whose model call fails for another reason, such as an outage that outlasts the retries, an exhausted quota or a refusal. The vendor's answer shows in the invocation record and under the run's app calls. Once the key is in, Replay from step on the run's page runs it again from read (Admin console: Runs). Known content, and a report without the post's text, never reach read: the rules decide those runs whatever the key.

pnpm policy:import reports the install as missing in a workspace without one, where a run that reaches read fails with unknown_install: create it as Installing a model app shows, named chatgpt. Every model app takes the same input and answers in the same shape, so read can call an install of claude or gemini instead, declared in installs with its app, or keep chatgpt with a fallback on a second install that answers while OpenAI is down (Falling back to another install).

What a reviewer sees

A moderation case in the queue the model's concern names, at critical when the model read that someone may be in danger, with the rule results, the model's answers under ai in the run context and the author's timeline of reports and strikes. The moderator's reject removes the content; decision.created tells your platform.

How to adapt it

Call your moderation classifier and image matching from the workflow, each through an http-api install for its API (Adding a vendor step), instead of sending their answers on the subject, add an appeal workflow reviewed by a different moderator, and export the audit trail for transparency reporting. Write the questions in your own community rules' words, move the 0.8 in triage once a backtest of past reports shows where your moderators and the model disagree, and create a queue for each concern (Admin console: queues and filters).

Run it

pnpm policy:import examples/policies/report-triage --publish

pnpm policies:check runs three fixtures. The model stub (scripts/ai-stub.mjs) answers for OpenAI, following the cue word in each post's text ([[stub:low]], [[stub:high]]); the pack carries none.

• expect.json: a post with a classifier score of 0.86, reported three times in the last 15 minutes, by an author with one strike, whose text mocks another user. The score rule fails, the model answers within_rules at 0.05, concern harassment and urgency high, and a moderation case opens in the harassment queue. The check removes the content (reject) and the run completes with a manual reject.
• expect-left-up.json: a disagreement about a TV show, reported once, that passes every rule and that the model reads as within the rules (0.95). It stays up without a case: an automated approve.
• expect-no-text.json: a report your platform sends without the post's text, with a classifier score of 0.91. The model is not asked, a case opens, and the check removes the content.