Hidden Resource in Robots.txt

Description

Hidden resources in robots.txt refer to sensitive paths or directories that are inadvertently exposed in the robots.txt file. The robots.txt file is used to instruct web robots on which parts of a website to avoid crawling or indexing. While intended for cooperation with search engine crawlers, disclosing sensitive paths can provide malicious actors with valuable information that could be exploited for unauthorized access or to identify potential attack vectors.

Recommendation

To mitigate the risk of exposing hidden resources in robots.txt, carefully review and sanitize the contents of the file to avoid revealing sensitive paths or directories. Ensure that only necessary and safe paths are included in the robots.txt file, and regularly review and update it to remove any inadvertently disclosed information.

References

• CWE-200
• Is your robots.txt file vulnerable? Here’s how to check and secure it
• OWASP 2021-A5
• Wikipedia: Robots exclusion standard