Bot economics
Bot economics is deciding crawler access based on whether a crawler can return a visit or a citation for the bandwidth it consumes.
Why it matters
Search crawlers send clicks, citation crawlers name sources, and training crawlers return neither. Grouping them together produces the wrong policy.
How it is measured
Read robots.txt and confirm each named agent has an intentional rule reflecting what it returns.
Checks that involve bot economics
Related terms
Questions
What is bot economics?
Bot economics is deciding crawler access based on whether a crawler can return a visit or a citation for the bandwidth it consumes.
Why does bot economics matter?
Search crawlers send clicks, citation crawlers name sources, and training crawlers return neither. Grouping them together produces the wrong policy.
How is bot economics measured?
Read robots.txt and confirm each named agent has an intentional rule reflecting what it returns.