Crawler policy

vape.deal operates an automated crawler that reads publicly listed prices from UK retailer websites. This page states how it identifies itself and how it operates.

User agent

Requests are sent with the user agent VapeDealBot/1.0 (+https://vape.deal/bot).

Rate

The crawler makes at most 1 request per second to any single host.

robots.txt

The crawler reads and follows each site's robots.txt before fetching any other page, and does not fetch paths it disallows.

When a bot user agent is rejected

Some hosts return an error to the VapeDealBot user agent while allowing crawling in their robots.txt. On those hosts only, the crawler retries with a browser-class user agent, still at 1 request per second, and still subject to that host's robots.txt.

Contact

For questions about this crawler, to request a change in how it accesses your site, or to submit a price feed directly, contact bot@vape.deal.