/robots.txt

Findings from Ariadne article crawl:

• Mean size of 427 bytes (15 lines)

• Range of sizes encountered: 0 to 101 lines

• On average, 21% of a file is ignored (ie. Consists of comments or blank lines)

• Errors found

Previous slide

Next slide

Back to first slide

View graphic version