docs: Block search engine indexing on subdomains (#11455)
## Summary
- Adds a dynamic `robots.txt` via Next.js Route Handler that blocks
search engine indexing on all subdomains (e.g., `v2-5-4.turborepo.com`)
- Subdomains are detected by checking if the host has more than 2 parts
- This only affects search engine crawlers - AI agents and users can
still access subdomain content
## Notes
This does not block actual access to the pages, only instructs search
engines not to index them.
