Skip to content
mondochat

Moderation

mondochat classifies text, photos, videos, voice notes and calls after they are delivered. A finding opens a case for your moderators in the dashboard. Nothing is hidden or removed unless your settings say so.

Before you start

Section titled “Before you start”
  • Adult content is allowed. Sex, nudity, erotic talk and selling sexual content between adults are never flagged, and there is no category for them.
  • You can sign in to the dashboard. Moderators work cases there. Admins edit settings and alert rules.
  • The kinds you want are on in your settings: moderation.text, moderation.photos, moderation.videos and moderation.voice are all on by default, and calls use moderation.calls.agora.

Steps

Section titled “Steps”

  1. Know what is classified.

    Kind What is read When
    Text The last 30 messages of the conversation, with no ids. A caption counts as a line. After delivery, at most once per conversation per minute.
    Photo A JPEG rendition of the photo. After delivery.
    Video The frames the sender’s browser sent with the upload, judged together. After delivery.
    Voice A transcript of the note, judged as the sender’s line in the conversation. After a delivered note is sent. A note held for consent waits until it is released.
    Call Agora call moderation per participant, or frames from the other participant’s client. While the call is active. See Calls.

    Reports from users open cases too, of kind report. See Reports, blocks and bans.

  2. Choose an action per category in moderation.rules. A category with no key acts as alert.

    Action What happens
    off The category is ignored.
    alert Opens a case, sends the moderation.flagged webhook and fires your alert rules. The content stays delivered.
    hold As alert, and the message or photo is hidden from the peer until a moderator acts.
    block As alert, and the message is redacted or the photo is rejected.

    The categories are:

    • Minors: any sign a participant is or could be under age, or any request for content of minors.
    • Trafficking: a third party controls a participant.
    • NonConsent: sextortion and sexual coercion, including threats to show images, and pressing on after a clear no.
    • Violence, SelfHarm, Hate, Harassment, Doxxing, Fraud: what the names say.
    • IllegalServices, Weapons, Drugs: selling or showing them.
    • ExternalPayment: the parties agree that money for the service changes hands outside your site. This is a business rule, not a safety one.
    • Political and Alcohol: images only.

    Most adult sites set Minors, Trafficking and NonConsent to block.

    { "moderation": { "rules": { "Minors": "block", "Trafficking": "block", "NonConsent": "block", "Fraud": "hold" } } }

    Send the whole settings document with PUT /v1/app/settings and If-Match; see Settings overview.

  3. Work the cases in the dashboard. A case has a kind (text, photo, video, voice, call or report), a severity (high, medium, low or info), categories, evidence, context messages and a timeline. A moderator can assign it, add notes and act on it. The actions depend on the kind:

    Kind Actions
    text, photo, video, voice dismiss, warn (a system card to the user), remove_content, ban
    call dismiss, warn, end_call, ban
    report dismiss, warn, ban

    Any other action for a kind answers 422. Acting on a case that is already decided answers 409 conflict with decided_by, decided_by_name and decided_at. The case list filters by status, kind, severity, category, assignee and age, and sorts by severity, newest or oldest.

  4. Receive moderation.flagged. The webhook is sent for every case the classifier opens, calls included. A report opens a case but sends report.created, not moderation.flagged. The data is {case_id, kind, severity, categories, action, subject}, where action is the rule that applied: alert, hold or block. It never carries message text or media.

  5. Get alerts in your own tools. An admin creates alert rules in the dashboard. A rule has one trigger and one destination.

    Trigger Fires
    moderation.flagged A new case of any kind except call, at or above a minimum severity and in the categories you pick (all, if you pick none).
    call.moderation A new call case, with the same filters.
    report.created A user reports someone.
    ban.created A ban is created.

    A new case fires exactly one trigger. A rule on moderation.flagged does not fire for a call case.

    Destination URL must be
    Signed webhook (Standard Webhooks, its own secret) Any https URL
    Slack hooks.slack.com
    Discord discord.com/api/webhooks or discordapp.com/api/webhooks
    Microsoft Teams *.api.powerplatform.com or *.logic.azure.com
    Email One or more addresses

    A URL on the wrong host for its type answers 422. Use Send test on a rule to deliver a sample alert and see the response code. A disabled rule delivers nothing. Deliveries retry and are logged like webhook deliveries; see Delivery and retries.

  6. Understand evidence. The flagged photo, frame, video, voice audio or transcript is kept while a case is open or actioned, and moderators open it through a link that lasts 120 seconds (1 hour for video, so it plays to the end). Evidence is deleted within 24 hours of an allow and 30 days after a dismiss.

Check it works

Section titled “Check it works”

The classifier is a real model, so no particular text is guaranteed to open a case on demand. Check the parts you control instead:

  1. Set Fraud to hold in moderation.rules and save the settings.
  2. Create an alert rule and press Send test. The destination gets a sample alert and the dashboard shows the response code.
  3. Have one test user report another with POST /v1/reports. A case of kind report opens in the dashboard, severity medium, and your report.created webhook and report.created alert rules fire.
  4. When a real message is flagged, a case opens in the dashboard and your endpoint gets moderation.flagged. With the rule on hold, the peer does not see the message until a moderator dismisses the case.

When it fails

Section titled “When it fails”
What you see Cause
No case for a message The category is off, the kind is off in moderation, or text was already read for that conversation in the last minute.
Case opens but nothing is hidden The rule is alert, which is the default for every category.
422 on a case action The action is not allowed for that kind.
409 on a case action Another moderator already decided the case.
422 saving an alert rule The URL host does not match the destination type.

Limits

Section titled “Limits”
  • Moderation runs after delivery. Nothing is held or blocked before a message arrives.
  • A call case from client frames never holds, blocks or ends the call, whatever your rules say.
  • A video is judged on the frames its sender’s browser sent, so a modified client can send clean frames.
  • A voice note is judged on its words only, and only when it is delivered.
  • If the AI provider refuses a photo as possible child abuse material, mondochat opens a Minors case even when Minors is off. Do not send that media to any AI and do not delete it. The photo is kept as evidence while the case is not dismissed, even if you set block or photos_days retention, or erase its uploader.
  • Alert payloads carry ids, category, severity, a short label and a dashboard link, never message text, media, media URLs, report reasons or reporter emails.
Section titled “Related topics”