Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
bolha.us is one of the many independent Mastodon servers you can use to participate in the fediverse.
We're a Brazilian IT Community. We love IT/DevOps/Cloud, but we also love to talk about life, the universe, and more. | Nós somos uma comunidade de TI Brasileira, gostamos de Dev/DevOps/Cloud e mais!

Administered by:

Server stats:

251
active users

bolha.us: AboutProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.7

#WebCrawlers

1 post1 participant0 posts today
Hacker News<p>Abusive AI Web Crawlers: Get Off My Lawn</p><p><a href="https://www.mythic-beasts.com/blog/2025/04/01/abusive-ai-web-crawlers-get-off-my-lawn/" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">mythic-beasts.com/blog/2025/04</span><span class="invisible">/01/abusive-ai-web-crawlers-get-off-my-lawn/</span></a></p><p><a href="https://mastodon.social/tags/HackerNews" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>HackerNews</span></a> <a href="https://mastodon.social/tags/AbusiveAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AbusiveAI</span></a> <a href="https://mastodon.social/tags/WebCrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>WebCrawlers</span></a> <a href="https://mastodon.social/tags/GetOffMyLawn" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>GetOffMyLawn</span></a> <a href="https://mastodon.social/tags/TechEthics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>TechEthics</span></a> <a href="https://mastodon.social/tags/InternetPrivacy" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>InternetPrivacy</span></a> <a href="https://mastodon.social/tags/AIRegulation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AIRegulation</span></a></p>
wolf of the wisp<p>got robots.txt ?</p><p>Dark Visitors - list of known AI (and other) agents on the internet : ‘the hidden ecosystem of autonomous chatbots and data scrapers’ <a href="https://darkvisitors.com/" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="">darkvisitors.com/</span><span class="invisible"></span></a> <a href="https://mastodon.ie/tags/DarkVisitors" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>DarkVisitors</span></a> <a href="https://mastodon.ie/tags/WebCrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>WebCrawlers</span></a> <a href="https://mastodon.ie/tags/WorldWideWeb" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>WorldWideWeb</span></a></p>
wolf of the wisp<p>got robots.txt ?</p><p>Dark Visitors - list of known AI (and other) agents on the internet : ‘the hidden ecosystem of autonomous chatbots and data scrapers’ <a href="https://darkvisitors.com/" rel="nofollow noopener noreferrer" target="_blank"><span class="invisible">https://</span><span class="">darkvisitors.com/</span><span class="invisible"></span></a> <a href="https://thefolklore.cafe/tags/DarkVisitors" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>DarkVisitors</span></a> <a href="https://thefolklore.cafe/tags/WebCrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>WebCrawlers</span></a> <a href="https://thefolklore.cafe/tags/WorldWideWeb" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>WorldWideWeb</span></a></p>
ExploreLive feeds
About