Various
This category includes various smaller crawlers and bots that individually account for a small share of web traffic but collectively represent a significant portion of crawling activity.
Current Rank
#5
Traffic Share
11.39%
Operator
Various
Category
Various
The 'Other Bots' category includes all web crawlers that individually don't make it into the top named crawlers list. This includes bots from smaller search engines, SEO tools, security scanners, content aggregators, and various other automated systems.
While each individual bot in this category contributes a small amount of traffic, together they can account for a significant percentage of all crawling activity on your website.
Common bots in this category include SEMrush, Ahrefs, MJ12bot (Majestic), DotBot, BLEXBot, and hundreds of other specialized crawlers.
Blocking all miscellaneous bots requires identifying and blocking each one individually in robots.txt. A common approach is to use a wildcard block with specific allows:
robots.txt
User-agent: Other Bots Disallow: /
Be careful when blocking bots broadly. Some crawlers, like those from SEO tools, help you monitor your site's performance. Others, like security scanners, help identify vulnerabilities.
Includes hundreds of different crawler user-agents
Mix of search engines, SEO tools, AI crawlers, and security scanners
Some may not respect robots.txt
Server-level blocking (IP, firewall rules) may be needed for persistent bots
Tools like Cloudflare Bot Management can help manage these at scale
Collectively, these bots can consume significant server resources. Monitoring your server logs to identify the most active crawlers and deciding which to allow or block is a good practice.
Some of these bots provide indirect benefits (SEO tool crawling helps you in those tools' databases), while others offer no value to your site.
LLM Pulse monitors how AI models mention your brand. See which crawlers visit your site and how that translates into AI visibility.