# Bing User-agent: bingbot Crawl-Delay: 3600000 Allow: / Disallow: /* # baidu User-agent: baiduspider disallow: / # Naver / LINE User-Agent: Linespider disallow: / User-Agent: YetiBot disallow: / # YaK/linkfluence.com User-Agent: linkfluence.com disallow: / User-Agent: YaK disallow: / # AdmantX User-Agent: admantx disallow: / # AhrefsBot User-Agent: AhrefsBot disallow: / # webmeup BLEX User-agent: BLEXBot Disallow: / # Buck User-agent: Buck Disallow: / # Clipper User-agent: Clipper Disallow: / # Common Crawl User-agent: CCBot Disallow: / # dotbot User-agent: dotbot Disallow: / # ExaBot User-agent: ExaBot Disallow: / # Femtosearch User-agent: FemtosearchBot Disallow: / # firstrain User-agent: http://firstra.in/FR3_0 Disallow: / # flamingosearch.com User-agent: Flamingo_SearchEngine Disallow: / # grapeshot User-agent: grapeshot Disallow: / User-agent: GrapeshotCrawler Disallow: / # NICT ICC User-agent: ICC-Crawler Disallow: / # Magpie Brandwatch User-agent: magpie-crawler Disallow: / # Mappy Crawler User-agent: Mappy Disallow: / # cyber.harvard.edu User-agent: mediawords Disallow: / # MJ12bot User-agent: MJ12bot Disallow: / # Nutch User-Agent: Nutch Disallow: / # comscore User-agent: proximic Disallow: / # Linguee User-agent: Linguee Disallow: / # Semrush User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / # sogou.com chinese search engine User-agent: Sogou web spider Disallow: / # toutia User-agent: ToutiaoSpider Disallow: / # Trendiction User-Agent: trendictionbot Disallow: / # trendkite User-agent: trendkite-akashic-crawler Disallow: / # Yandex User-agent: Yandex Disallow: / # ZoominfoBot, not related webmeeting system: zoom User-agent: Zoominfo Disallow: / # Punkspider vulnerability scanner User-agent: Punkspider Disallow: / User-agent: punkspider.org Disallow: / # WebDataStats, russian vulnerability scanner User-agent: WebDataStats Disallow: / User-Agent: * Allow: /pc/ Allow: /sp/ Allow: /campaign/ Allow: /rss/ Disallow: /unsubscribe/ Sitemap: https://dietnavi.com/sitemap.xml