indexo.dev

crawlee.dev

.dev crawl

First seen 2026-04-13 · Last seen 2026-05-01 · ok HTTP/1.1 200 3254 ms crawled 2026-05-05

US · 13.33.235.112 · AS16509 Amazon.com, Inc.

Reputation 100/100

sector tech type docs

HTML metadata

Title
Crawlee · Build reliable crawlers. Fast.
Description
Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.
Language
en
Generator
Docusaurus v3.10.0
Canonical
https://crawlee.dev/
Translations
  • en
Feeds

Open Graph

url
https://crawlee.dev/
title
Crawlee for JavaScript · Build reliable crawlers. Fast.
locale
en

Technology

CDN
Amazon CloudFront
Server
nginx
Analytics
  • Google Tag Manager

Third-party hosts loaded (1)

  • www.googletagmanager.com×1

Social

DNS records live

NS
  • ns-1067.awsdns-05.org
  • ns-1761.awsdns-28.co.uk
  • ns-330.awsdns-41.com
  • ns-919.awsdns-50.net
TXT
  • google-site-verification=xHLQTtwKXyGWT5422wLDLRmFP_IwehLQRiKHJXKWiN0

Certificate (current)

Amazon RSA 2048 M04
from 2026-05-04 to 2026-11-18
Expires in 182 days

Full certificate history on crt.sh ↗

HTTP security headers

Header hygiene 60/100 Checked live page: https://crawlee.dev/

present
  • strict-transport-security
  • x-frame-options
findings
  • missing Content Security Policy
  • missing content type protection
  • missing Referrer Policy
  • missing Permissions Policy
Header values
x-frame-options
SAMEORIGIN
strict-transport-security
max-age=31556952

Links to (7)

Linked from (1)