User-Agent: *
Allow: /
Disallow: /admin
Disallow: /bookmarks
Disallow: /search

Sitemap: https://techcruncher.com/sitemap.xml