User-agent: * Crawl-Delay: 30 Disallow: /.well-known/* Disallow: /*?*category= Disallow: /*?*tag= Disallow: /*?*podcast= Disallow: /*?*blog= Disallow: /*#type-blog Disallow: /*#type-podcast User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Twitterbot Crawl-Delay: 0