robots.txt

A robots.txt file tells crawlers what they're allowed to do on a hostname. On redirect.pizza you choose which robots.txt is served for your redirected hostnames, at team level or per hostname. A setting on team level is inherited by every hostname unless you override it. Available from the Basic plan and up.

robots.txt options in the team settings

Options

No robots.txt (default)

We don't serve a robots.txt. A request for /robots.txt is not a special case: it's matched and redirected like any other path on your hostname. With path forwarding enabled, a crawler ends up at the destination's robots.txt. Crawlers treat a missing robots.txt as "everything is allowed".

Explicitly allow all

We serve a robots.txt that allows all crawling. The effect is the same as the default, but explicit:

User-agent: *
Disallow:

Disallow bad bots

We serve a robots.txt that disallows a long list of known bad bots: crawlers that generate traffic and clutter your analytics without any benefit to you. Good bots such as Googlebot and Bingbot are unaffected. This option can reduce the number of hits on your account.

The list is based on the Apache ultimate bad bot blocker. You can see the resulting file at redirect.pizza/robots.txt.

Disallow all

We serve a robots.txt that tells crawlers not to crawl the hostname at all:

User-agent: *
Disallow: /

Custom

Need your own robots.txt? That's possible with a dedicated IP, because the custom file is served from your own IP address on our edge network. Contact sales for more information.

Good to know

robots.txt is a request, not an enforcement. Well-behaved crawlers honor it; some bad bots ignore it. For traffic you actively want to block, see the Web Application Firewall.

Which option is right for you depends on your SEO strategy. When in doubt, ask your SEO specialist.

More articles

Setup & Configuration
What are these DNS changes?
What DNS types can I use?
How long does the DNS change take?
Troubleshooting new redirects
Redirecting non-www to www
Redirect settings
Matching
Regex matching
Wildcard subdomains
Destination variables
Dynamic destinations
Redirect flattening
SSL profiles
robots.txt
Monitoring redirect.pizza itself
Features
Automatic HTTPS
Automatic DNS
Nameservers
Analytics
Broken destination monitoring
Email forwarding
Pause and resume
Dedicated IP
API
MCP
Zapier
Terraform provider
Security & Account
Team management
Two-factor authentication
SAML SSO
SSO via Okta
Summary email
HSTS (HTTP Strict Transport Security)
Prevent foreign embedding
Web Application Firewall (WAF)
Security & compliance
Subprocessors
Service level agreement (SLA)
Walkthroughs
GoDaddy: Forwarding with HTTPS support
Namecheap: Forwarding with HTTPS support
Cloudflare: Forwarding with HTTPS support
Network Solutions: Forwarding with HTTPS support
Hover: Forwarding with HTTPS support
Amazon Route 53: Forwarding with HTTPS support
Google Cloud DNS: Forwarding with HTTPS support
Azure DNS: Forwarding with HTTPS support
DigitalOcean: Forwarding with HTTPS support
NS1: Forwarding with HTTPS support
DNS Made Easy: Forwarding with HTTPS support
TransIP: Forwarding with HTTPS support
EuroDNS: Forwarding with HTTPS support
Mijndomein: Forwarding with HTTPS support
Freenom: Forwarding with HTTPS support

Domain redirects delivered hassle-free

Get started right away

  • Free plan
  • No creditcard required

redirect.pizza has all your favorite ingredients

Analytics

Who is still visiting those old domains? Gain valuable insight with detailed statistics and find out where the hits are coming from.

Learn more

API/MCP

Connect your software or AI clients to create redirects via our API and MCP. Automation makes your life easier.

Learn more

Destination Variables

You control how we handle your destination URL. Want to include your domain extension as a query string parameter? No problem.

Learn more

Dynamic destinations

Steer your traffic to different destinations, based on rules you define.

Learn more

Path forwarding

Keep the original path and keep it in the destination for a smooth domain migration and keeping your SEO game intact.

Learn more

Query Parameter Forwarding

Forward query parameters found on the source URL to the destination.

Learn more

Multiple Redirect Types

Choose the status code that works for you. Choose 301/307 (permanent), 302/308 (temporary) or frame your destination.

Learn more

Team management

Doing things all by yourself isn't much fun. Invite your co-workers to manage the redirects and work as a team.

Learn more

Automatic DNS

Integrate directly with your DNS provider, and apply the DNS changes from one platform.

Learn more

QR code generator

Create a QR code that you can use for marketing purposes. You can change the destination anytime after printing.

Learn more

Import & Export

Manage your redirects without losing precious time. Import and export redirects in our UX-friendly interface.

Global edge network

Redirecting in a blink of an eye; processing millions of request per hour in milliseconds.

Learn more

Broken destination monitor

Automatically checking your destinations to ensure it's reachable for everyone.

Learn more

Privacy First

We're based in Europe and are fully compliant with GDPR. Privacy-first!

Learn more

Wildcard subdomains

For when your domain utilizes a lot of different subdomains. Cover all subdomains with a single redirect.

Learn more

Tags

Group your redirects by adding tags. Keep everything organized and easy to find.

Bulk actions

Want to apply the same action for multiple redirects in one go? Do yourself a favor and start using bulk actions.

Dark mode

Give your eyes some well deserved rest by using the dark mode. Sync with your system or choose manually.

HSTS

Enable HSTS (HTTP Strict Transport Security) on team or domain level for an extra layer of protection.

Learn more

Change Destination Anytime

Once you're set up, you can change the destination URL anytime you want, and see the change in real-time.

Two-Factor Authentication

Secure your account with TOTP two-factor authentication. Scan the QR code, setup your app of choice, and you're safe(er).

Learn more

IDN Support

Add international hostnames with non-Latin characters. We ❤ that.

Support

Chat away! We're happy to help set up your account and answer your questions.