AI crawler

In practice

A useful distinction: training collectors build the long-term memory of models; search agents decide citations in real time. Blocking the first category on principle is debatable; blocking the second amounts to opting out of responses. Log audits reveal who actually visits — and at a frequency that often surprises.