|
Filling in the below fields will directly generate your robots.txt file in the rightmost last box.
- Use an empty line to create an empty Disallow field, which means: allow the robots everywhere.
- The locations should begin with a slash and they must end with a slash, if it's a directory.
- To allow a robot everywhere, place an empty line in the field. (An empty Disallow line stands for Allow everywhere)
Tip: to generate a list of files, consider the following GNU commands:
find -printf "/%P\n"
find -type f -printf "/%P\n"
find -name *.gif -printf "/%P\n"
find -type f -printf "/%P\n" |grep google
|