Log analysis
Log analysis uses server logs to observe the real behaviour of crawling robots: pages visited, crawl frequency, and HTTP response codes encountered. It provides the only exact view of the crawl, without sampling.
In practice
It answers questions no external tool can settle definitively: is Googlebot wasting time on parameterised URLs? Are strategic pages visited weekly or quarterly? On large volumes, cross-referencing logs, sitemap, and audit data pinpoints both waste and blind spots with surgical precision.