Automation score (human vs. script)

The automation score rates how script-driven vs. human-driven each user’s API traffic is, on a 0.0–1.0 scale:

• HIGH (→ 1.0) — traffic is mostly driven by automatic scripts / batch jobs / cron.

• LOW (→ 0.0) — a human is using the service interactively (a chat UI, or a human-driven coding agent such as Claude Code).

It is a heuristic for triage, not a verdict. Always read it together with the reported confidence and the per-signal breakdown.

Note

This feature profiles individual users’ traffic patterns. Computing a score reads that user’s api_logs rows over the window — when they sent requests, how long their prompts were, which client they used, and precomputed shape statistics of their newest user message (length, character entropy, and a hash — see signal 7). It is reachable only from the admin endpoints and the operator CLI below, never by the user themselves and never from any public route. A deployment that operates this on real traffic should say so in its own privacy policy, and should treat a score as an input to a human decision rather than as grounds for automated enforcement.

Inputs

The score is computed over a trailing window (default 30 days; the admin endpoints accept days in 1..90). Only api_logs rows with a non-null user_id are considered — anonymous traffic is excluded. For each user the following per-user aggregates are gathered:

• request count N, and n_chat = rows where num_user_turns IS NOT NULL (chat-style requests; NULL for embeddings / raw completions);

• one-shot count (num_user_turns = 1) and the p90 of num_user_turns;

• tool-call counts (num_tool_calls IS NOT NULL, and > 0);

• the 25/50/75th percentiles of positive prompt_tokens, and n_sz = the number of rows with prompt_tokens > 0 (the availability count for the prompt-size signals);

• the share of requests carrying a coding-agent opener (metadata->>'agent' present — see the agent_opener_override signal below);

• a request-weighted breakdown of metadata->>'user_agent';

• a UTC hour-of-day histogram (24 buckets);

• the 25/50/75th percentiles of the inter-arrival gaps between consecutive requests.

The signals

The score blends seven signals. Each maps its raw metric to an automation sub-score in [0, 1] (HIGH = looks automated), and has a default weight. The four signals the feature was originally specified around are user turns, turn length, user-agent, and daily activity; two small supporting signals disambiguate the main confounder (a high-volume but human-driven coding agent); and user_message_shape measures the user’s own messages specifically.

Signal	Axis	Default weight	Available when
turn_pattern	user turns	0.24	n_chat ≥ 5
prompt_size_dispersion	total prompt size	0.17	n_sz ≥ 8 and median > 0
user_message_shape	user-message size & entropy	0.15	≥ 1 part below
client_tool_prior	user-agent	0.16	always (≥ 1 request)
daily_activity_shape	daily activity	0.27	≥ 1 timing part below
tool_call_human_tell	(support)	0.08	n_chat ≥ 5 and some tool use
agent_opener_override	(support)	0.08	agent_share ≥ 0.05

A signal that lacks enough data for a user is dropped, and the remaining weights are re-normalized over the survivors — a missing signal is never imputed as 0 (which would falsely pull the score toward “human”). The weights need not sum to 1.0 (they total 1.15); the runtime always divides by the available weight. A user whose requests all predate the user-message columns simply lacks user_message_shape: the other six signals blend as usual, and confidence is slightly lower.

1. turn_pattern — user turns

An interactive session resends a growing history, so num_user_turns climbs 1, 2, 3, …; a script firing independent one-shot completions emits num_user_turns = 1 on essentially every request.

f1          = one_shot_chat_requests / n_chat          # fraction stuck at 1 user turn
depth_factor = 0.5 if p90(num_user_turns) >= 3 else 1.0
sub          = clamp01(f1 * depth_factor)

The depth_factor halves the sub-score for a user who has demonstrably held deep (p90 ≥ 3) multi-turn threads, so a human who also fires many one-shot requests is not branded a script.

2. prompt_size_dispersion — length of user turn

This uses prompt_tokens, which (OpenAI semantics) is the total input for the request — the system prompt + the whole conversation history resent that turn + tool definitions + the latest user message — not the user message in isolation (api_logs has no per-role token breakdown). It is the only available proxy for “length of user turn”. Templated automation assembles each request from a fixed template, so its total prompt size clusters tightly; an interactive human’s requests vary widely (a one-word follow-up, then a long paste). The discriminator is therefore the robust relative dispersion (IQR / median) of that total size, which is scale-free and resistant to a single huge pasted prompt:

rcv = (p75 - p25) / median        # over positive prompt_tokens
sub = clamp01(1 - rcv / 0.5)      # rcv >= 0.5 -> 0 (human-varied); rcv = 0 -> 1 (templated)

3. client_tool_prior — user-agent

The User-Agent is the most spoofable signal, so it is a soft, low-weight prior, never decisive. Each request’s UA is classified into a client class (a Python port of the frontend parseClientTool), with a per-request automation value:

Client class	Examples	Value
Interactive / coding agent	claude-code, cline, cursor, codex, browser	0.10
Ambiguous SDK	openai-python, openai-node, anthropic-python/-sdk	0.50
Raw HTTP library / API tool	python-requests, httpx, aiohttp, curl, wget, okhttp, axios, go-http, postman, …	0.85
Unknown (recognized leading token)	myagent/1.0	0.60
Absent / unrecognized UA	—	0.70

ua_base = request-weighted mean of the per-request class values
sub     = clamp01(ua_base * (1 - 0.85 * min(agent_share, 1)))

SDKs sit at a neutral 0.5 because a human chat UI can sit on top of openai-python. The agent_share term lets the coding-agent opener (see below) pull the prior toward human even behind a scripty UA.

4. daily_activity_shape — daily activity

The behavioral fingerprint hardest to fake at scale. Up to four parts are fused and re-normalized over whichever pass their data floors. Each is HIGH = automation:

Part	Weight	Formula	Floor
Hour coverage	0.20	clamp01((coverage - 0.5) / 0.5), coverage = distinct active UTC hours / 24	N ≥ 10
Hour entropy	0.20	clamp01((Hnorm - 0.5) / (0.92 - 0.5)), Hnorm = ShannonEntropy(hours) / log2(24). The 0.92 is the ceiling at which this part saturates, a tuning constant	N ≥ 10
Nightly rest gap	0.30	clamp01(1 - max_quiet_gap_hours / 6)	N ≥ 10
Inter-arrival regularity	0.30	clamp01(1 - gap_rcv / 1.0), gap_rcv = (p75 - p25) / median of gaps	≥ 3 gaps

max_quiet_gap_hours is the longest circular run of inactive clock-hours — a human’s nightly sleep gap pushes the rest-gap part toward 0, while 24/7 operation leaves no gap (→ 1). The timezone-invariant parts (regularity, rest gap, entropy) carry most of the weight, so an unknown user timezone shifts the histogram without distorting the verdict (hours are bucketed in UTC).

5. tool_call_human_tell — supporting

Agentic tool use (num_tool_calls > 0, from OpenAI tool_calls / Anthropic tool_use blocks) is the fingerprint of a human-driven coding loop. It is a one-directional human tell: it is dropped entirely when there are no tool calls (absence of tool use is not evidence of automation), and otherwise can only pull the score toward human, never raise it:

sub = clamp01(0.5 - toolcall_share)   # share>=0.5 -> 0 (strongly human); share~0 -> ~0.5 (neutral)

6. agent_opener_override — supporting

metadata->>'agent' is a coding-agent identity parsed from the system-prompt opener ("You are Claude Code, …"). Because it is content-derived (not a header), it is hard to fake incidentally and is treated as strong human evidence. It is available only when the opener appears on ≥ 5% of requests, so it can only pull toward human; its absence is uninformative:

sub = clamp01(0.15 - agent_share)     # opener pervasive -> ~0 (human)

It also drives the hard human clamp in the combination step.

7. user_message_shape — user-message size & entropy

Where prompt_size_dispersion looks at the whole input, this signal looks at the user’s own messages. Three properties of the newest user-role message per request are precomputed at log time (in user_message_stats, alongside conversation_shape) as cheap columns — char length, Shannon character entropy (bits/char), and a stable 64-bit hash of the stripped text — so the score never de-TOASTs the prompt. Three parts are fused and re-normalized over whichever pass their floors, each HIGH = automation:

Part	Weight	Formula	Floor
Size dispersion	0.40	clamp01(1 - size_rcv / 0.5), size_rcv = (p75 - p25)/median of last_user_msg_chars	≥ 8 sized msgs
Per-message entropy	0.25	clamp01(1 - mean_entropy / 4.0) (natural language ≈ 4 bits/char)	≥ 5 msgs
Cross-message repetition	0.35	clamp01((1 - distinct_ratio) / 0.5), distinct_ratio = distinct(hash)/count	≥ 8 hashed msgs

Templated automation sends near-constant-length user messages (low size dispersion), low-entropy structured payloads, and resends the same message over and over (low distinct ratio → high repetition); an interactive human varies all three. Because the columns are populated only for new traffic, the signal is simply unavailable (dropped) for users whose requests all predate the migration.

Because it measures only the user’s own messages, a script can look human here only by actually varying what it sends.

Combining the signals

A          = signals available for this user
weight_sum = sum(weight_i for i in A)
raw        = sum(sub_i * weight_i for i in A) / weight_sum      # re-normalized blend

# Hard human clamp: a high-volume coding-agent user with a real nightly rest gap
# can never be branded above "mixed" on volume alone.
if agent_share >= 0.3 and rest_gap_part_available and rest_gap_score < 0.5:
    raw = min(raw, 0.5)

# Confidence shrinkage toward the neutral 0.5 prior for low-volume users.
alpha = N / (N + 30)
score = clamp01(alpha * raw + (1 - alpha) * 0.5)

# coverage = available weight / total signal weight, so confidence stays in [0,1].
confidence = alpha * (weight_sum / TOTAL_WEIGHT)

• Re-normalization (raw) makes the blend depend only on the signals that actually had data, so a pure-embeddings batch user — whose num_*_turns columns are all NULL — is still scored on its user-agent and daily-activity shape.

• Shrinkage (alpha = N / (N + 30)) pulls users with little traffic toward the neutral 0.5 prior. Data and prior weigh equally at N = 30 (alpha = 0.5), and the data dominates above that; at N = 5, alpha ≈ 0.14 (the score is pulled ~86% to 0.5).

• confidence combines the request volume (alpha) with the share of the total signal weight that was actually available (weight_sum / TOTAL_WEIGHT), so sparse verdicts — and users missing the newer signals — read as lower-confidence.

• Users with N < 5 requests are flagged insufficient_data.

Bands

The final score maps to a band label (advisory — always read with confidence):

The lower bound is inclusive and the upper bound exclusive (SCORE_BANDS is scanned top-down with score >= lower), so 0.35, 0.60 and 0.80 each fall in the band above:

Band	Range
likely_human	0.00 ≤ s < 0.35
mixed_or_uncertain	0.35 ≤ s < 0.60
likely_automated	0.60 ≤ s < 0.80
scripted_batch	0.80 ≤ s ≤ 1.00

Caveats

• It’s a heuristic. No single signal is decisive; each is individually spoofable, so the user-agent is weighted low and behavioral signals dominate. Treat the score as triage, not proof.

• Timezone. Hour-of-day is bucketed in UTC; the design leans on the timezone-invariant daily-activity parts so a non-UTC human is not mislabeled as night-active, but a genuinely multi-timezone account can inflate hour coverage.

• Shared / role accounts. Shared or team accounts blend human and script traffic into a mid “mixed” score (correct). internal / admin accounts may legitimately run automation — the score stays role-agnostic, so exclude them from any automated enforcement and read the score with their role in mind.

Reproducing it from the CLI

# rank the most script-like users in the last 30 days
python ops/db/analysis/user_automation_score.py --min-requests 20

# full per-signal breakdown for one user
python ops/db/analysis/user_automation_score.py --email [email protected]

Code map

Layer	Location
Core scoring + gather SQL	apps/backend/serving/analytics/automation_score.py
Store methods	LogStore.get_user_automation_score / get_bulk_user_automation_scores (apps/backend/serving/storage/postgres_log.py)
Admin endpoints	GET /admin/users/{user_id}/automation-score, GET /admin/users/automation-scores (apps/backend/serving/servers/routers/admin/users.py)
Admin dashboard	per-user button + bulk “Automation” column in the Users tab
CLI	ops/db/analysis/user_automation_score.py

The scoring methodology and the SQL that gathers per-user aggregates live in one module (serving.analytics.automation_score), shared by the admin endpoints and the CLI, so the two can never drift. The pure scoring functions have no database dependency and are unit-tested in tests/unit/test_user_automation_score.py.