Hidden Resource in Robots.txt

Description

The robots.txt file specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. Not all robots cooperate with the standard; email harvesters, spambots, malware, and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. ^Wikipedia

Recommendation

Do not reveal paths in robots.txt file.

References

• Is your robots.txt file vulnerable? Here’s how to check and secure it
• Wikipedia: Robots exclusion standard
• CWE-200
• OWASP 2007-A6
• OWASP 2021-A1