Robots.txt is a small text file at the root of your website that tells search engine crawlers which parts of the site they may visit and which to leave alone. It is the first thing a crawler reads when it arrives. Most sites need only a few lines in it, and one wrong line can hide the whole site.
It controls visiting, not ranking, and it is a request rather than a lock. A blocked page can still appear in results if other sites link to it, so it is not a way to keep anything private.
Plain text at /robots.txt, parsed per the Robots Exclusion Protocol: User-agent groups with Allow and Disallow rules, plus Sitemap: lines. Rules match URL path prefixes, with * and $ wildcards. Disallow blocks crawling, not indexing; use a noindex meta tag or header for pages that must not appear in results, and do not block those pages in robots.txt or the tag is never seen.
The classic failure is a staging rule carried to production: Disallow: /. AI crawlers have their own user agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) and many sites now block them by default through a plugin. Decide that deliberately: blocking them removes you from AI answers as well as from training.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml