Crawler Log Triage

Name: Crawler Log Triage
Author: pathwren

Paste access-log lines: every bot named, fakes flagged, robots.txt and WAF rules. Free, no key. Direct endpoint, no gateway: https://www.pathwren.workers.dev/c/smithery-direct/mcp/triage (streamable HTTP, no key, no signup). Tools: triage_log, find_impersonators, summarize_by_operator, robots_from_log, waf_ruleset_from_log, example. Call the `example` tool with no arguments for a worked answer against real data before you write any code. What it does and how, in one page: https://www.pathwren.workers.dev/c/smithery-direct/mcp-triage.html

该来源不提供完整文件导出（国内平台多为平台内托管），仅存元数据与原链

模型生成摘要（rules/v1 · 2026-09-27 16:22）：Paste access-log lines: every bot named, fakes flagged, robots
这是模型对公开材料的总结，不是官方声明，请以原链内容为准。

接入信息

传输形态: http
鉴权方式: 鉴权未知
端点: https://crawler-log-triage--pathwren.run.tools

鉴权方式未标注，请核对官方文档后再接入——不要直接使用以下片段

{
  "mcpServers": {
    "Crawler Log Triage": {
      "url": "https://crawler-log-triage--pathwren.run.tools"
    }
  }
}

能力清单

工具	说明
triage_log	Paste log lines — combined, JSON, `uniq -c`, `<ip> <ua>` or bare user-agents, auto-detected — for per-line crawler, operator, purpose and cost of blocking, a rollup by operator and category, and every line whose address contradicts its claim. Log text, never a URL. Example: log='66.249.66.1 Googlebot/2.1' returns Googlebot, Google, search, verified.
find_impersonators	Only the lines claiming a crawler whose operator publishes address ranges, from an address in none of them — 1984 IPv4 and 1062 IPv6 prefixes, 15 sources. Reverse-DNS operators come back with the command to run: this server makes no outbound request. Example: log='203.0.113.9 Googlebot/2.1' returns one impersonation.
summarize_by_operator	Aggregate only: who crawled you, how many requests each, what share, which category, and what blocking each would cost. Eats a `uniq -c` table straight from a shell pipeline. Example: log='412 GPTBot/1.2' returns OpenAI, 412 requests, 100%, ai-training.
robots_from_log	A robots.txt naming only the crawlers in your log, each with its request count and cost of blocking, plus a warning for any that do not documentably obey it — there the file is a request, not enforcement. Example: log='412 GPTBot/1.2', stance='block-ai-training' blocks GPTBot only.
waf_ruleset_from_log	nginx, Caddy, Cloudflare, HAProxy or Apache rules for only the crawlers in your log. The reply warns that a UA rule stops only an honest client, and that impersonation is an address problem needing the published prefixes as an allowlist. Example: log='412 GPTBot/1.2', target='nginx', scope='ai-training'.
whoami	Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and your own request triaged as if it were one line of an access log — the same parser, the same index, the same verification against operator-published prefixes that triage_log applies to a whole file. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before.
example	Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — triage_log on three lines in three of the five formats this parser accepts: an address Google publishes, an address in no published range claiming to be the same crawler, and a `uniq -c` count — all of it constants in this file, checked against our own mirror, with nothing fetched from anyone — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs triage_log with {"log":"66.249.66.1 Googlebot/2.1\n203.0.113.9 Googlebot/2.1\n412 GPTBot/1.2"} and returns its real answer.

纠错与举报（发现条目失效、署名有误或涉及侵权？）

提交举报 / 纠错

侵权举报经核验成立后，我们会即时下线该条目并删除已存的内容副本。