A crawler instruction does not require a browser or unauthorized visitor to stay out. Private content needs authentication and authorization, not merely a robots rule.
Start with the evidence
Open protected routes without signing in and inspect their actual responses using approved tests. Confirm that sensitive content is not included in publicly served HTML or data files.
Make the change deliberately
Protect private resources server-side. Use crawler controls for crawl policy separately, and never describe a page as secure solely because a robots file disallows it.
Further reading
Platform guidance can change. Consult the current official documentation before making platform-specific changes.
Have a question about this?
Ask Benchmark about your situation. Your inquiry will include this article's topic, and any paid work starts with an agreed scope.
Ask an Expert