KokoSEO

Robots.txt Tester & Generator

Test your robots.txt rules, or build one from scratch.

Technical SEO

Test up to 20 URLs at once against a site’s robots.txt and see exactly which rule decides each one. Below it, build a fresh robots.txt by ticking the bots you want to keep out — including AI crawlers such as GPTBot and ClaudeBot — then test the result on the same page without leaving it.

Satu URL per baris, maksimal 20. Yang dicocokkan bagian path-nya.

Leave empty to fetch robots.txt straight from the site, or build one with the generator below.

robots.txt generator

Tick the bots you want kept off your site. The result assembles instantly below, and can be tested straight away with the tester above.

Search engines

Blocking anything here removes your pages from their search results. That is almost never what you want.

AI crawlers

They take your content to train on or answer with through a language model. Blocking them does not affect search rankings.

SEO tool crawlers

Competitors use them to dissect your links and keywords. Blocking them also removes your own data from those tools when you use them yourself.

Social previews

They fetch the title and image when your link is shared. Blocking them makes the link appear bare, with no picture.

One per line. You may use * and $ exactly as real robots.txt rules do.

Paths that stay crawlable even though they sit inside a forbidden one.

Leave empty if not needed. Google ignores it.

One URL per line.

  • No Sitemap line yet. Adding one is the cheapest way to help crawlers find every page you have.

Result

# robots.txt
# Dibuat dengan KokoSEO — https://kokoseo.com/tools/robots-tester
#
# Ingat: robots.txt mengatur perayapan, bukan pengindeksan. Halaman yang
# dilarang di sini masih bisa muncul di hasil pencarian kalau ada situs lain
# yang menautkannya. Untuk benar-benar menyembunyikan, pakai meta noindex.
#
# Berkas ini juga bersifat publik dan sekadar imbauan — perayap jahat
# mengabaikannya. Jangan pernah mencantumkan jalur rahasia di sini.

User-agent: *
Disallow:
# Baris kosong di atas berarti: semuanya boleh dirayapi.

Save it as robots.txt at the domain root so it opens at https://yoursite.com/robots.txt. Crawlers do not read it anywhere else.