robotstxt.com

.com crawl

First seen 2026-06-01 · Last seen 2026-06-02 · ok HTTP/1.1 200 301 ms crawled 2026-06-01

US · 104.21.36.107 · AS13335 Cloudflare, Inc.

Reputation 94/100 dmarc monitor-only

Classifying

HTML metadata

Title
robots.txt Tools & Resources
Description
Tools and guides to manage and optimize your robots.txt file. Created by ex-Google engineer Fili.
Canonical
https://robotstxt.com/

Technology

CDN
Cloudflare

Registration

Registrar
Hosting Concepts B.V. d/b/a Registrar.eu
Created
2000-06-06
Expires
2032-06-06 2193 days left
Updated
2026-04-21
Name servers
  • chan.ns.cloudflare.com
  • sri.ns.cloudflare.com

DNS records live

NS
  • chan.ns.cloudflare.com
  • sri.ns.cloudflare.com
MX
  • 1 aspmx.l.google.com
  • 10 aspmx2.googlemail.com
  • 10 aspmx3.googlemail.com
  • 5 alt1.aspmx.l.google.com
  • 5 alt2.aspmx.l.google.com
Verified for
  • Google

Email authentication partial

SPF
v=spf1 include:u2184761.wl012.sendgrid.net include:_spf.google.com ~all
softfail (~all)
DMARC
v=DMARC1; p=none; pct=100; rua=mailto:dmarc@fili.com
policy: none (monitoring only)
DKIM
  • google: v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs5dbBs19XVc7kVFjImCkczM18kDgNyQeM7cklrOtg34r7bnaP0Wk8GhEB9oZ4IgucyMdUIiV9zu2jp…
  • s1: k=rsa; t=s; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAz6s67f4bChMERR2bh6b63pjIYswx9QNHJj+RaaVmmJA2HIDp1vb4PbQLPkn76ve9jj1u4+dQZvqgcjJxLH…
  • s2: k=rsa; t=s; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDwbcr9li8SbE9bsV8kKW6izv/IKPcK3LUhVL8wmRkWXTHafu45iKuhJxEg2GQDQUpkzdmrm/d35XKg2m+QCBz2vp…
selectors probed

Certificate (current)

E7
from 2026-04-21 to 2026-07-20
Expires in 45 days

HTTP security headers

Header hygiene 70/100 Checked live page: https://robotstxt.com/

present
  • strict-transport-security
  • x-frame-options
  • x-content-type-options
findings
  • missing Content Security Policy
  • missing Referrer Policy
  • missing Permissions Policy
Header values
x-frame-options
SAMEORIGIN
x-content-type-options
nosniff
strict-transport-security
max-age=63072000; includeSubDomains; preload

Links to (7)

Linked from (7)