Who we are and why we visit
HealthAgencies.co.uk is an independent site that compares UK healthcare recruitment agencies. To compare every agency the same way, our crawler reads each agency's own public website. It is not used to train AI models.
What we read
- Your public pages: home, services, specialties and similar.
- Your job listings and job adverts, so we can see how many are live and how up to date they are.
- Your sitemap and robots.txt.
We never log in, fill in forms or apply for jobs, and we don't download images, fonts or video. We keep a copy of the pages we read, so we can show you exactly what we saw if you query a result.
How politely
- Mostly overnight, one request at a time, with a pause between requests.
- After the first visit we only come back on a schedule: job listings about daily, individual adverts about weekly, other pages about monthly.
- We ask your server whether a page has changed before downloading it again, so unchanged pages cost almost nothing.
- We follow your robots.txt, including
Crawl-delay. - We never try to get past bot checks or CAPTCHAs. If your site blocks us, we stop.
How to recognise it
- User agent:
HealthAgenciesBot/1.0 (+https://healthagencies.co.uk/about/bot) - It only comes from this IP address:
13.42.169.219(current list: /bot/ips.json). - Its reverse DNS name ends in
.bot.healthagencies.co.ukand points back to the same IP address, so your firewall can check it is really us.
Let us in
Security tools often block automated visitors by default. If yours blocks us, we can't read your site, so we can't include what's on it when we compare agencies. Here's how to allow our crawler in Cloudflare, WordPress and other hosts.
Slow us down or keep us out
Add this to your robots.txt. Crawl-delay is the number of seconds between our requests:
User-agent: HealthAgenciesBot Crawl-delay: 5 Disallow: /private/
To keep us out completely, use Disallow: /, or just email us and we'll stop. Either way we'll respect it, but then we can't read your site for our comparisons.
Contact us
Email bot@healthagencies.co.uk. Agencies can also report an error or use their agency account.