Crawler FAQ - Meet zbot!
-
How does Zvents get all this information?
Our friendly crawler, zbot, goes out and explores the web to find information that's relevant to our products. If there's a match, we index the information, and it becomes part of our index. Much of our information is also submitted by users, who want to promote their events and businesses. The following information should answer any questions you may have about how this works. But if you have further questions, please contact Zvents. -
I want Zvents to crawl my site. What can I do?
If you have a website where you list events, Zvents may be able to automatically get the event information directly from your site through our automated process. If you would like Zvents to crawl your website, submit the URL here. Crawling of your site is subject to a review by our content staff, and not all sites are indexable by Zvents. If you want to get your listings in Zvents quickly, the fastest way is to enter the information yourself.
-
What is the name of the Zvents crawler?
zbot
-
What user-agent does the Zvents crawler use?
The Zvents crawler is implemented using the Heritrix open source crawler, developed by the Internet Archive. The user-agent string we currently use is the following:
Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.zvents.com/; +zbot)
-
How often will zbot visit my website?
zbot will visit your website no more than once every few seconds, on average. Sometimes we run multiple instances of zbot on different machines, in which case, we scale the host delay factor (politeness interval) appropriately to keep the visit rate to an average of once every few seconds.
-
How can I prevent zbot from crawling all or part of my site?
Our crawler obeys the Robots Exclusion Standard. To prevent our crawler from fetching any pages on your site (except /robots.txt of course), add the following lines to your /robots.txt file:
User-agent: zbot
Disallow: / -
Why is zbot trying to fetch a page from my site that does not exist?
zbot automatically discovers URLs by fetching HTML pages and extracting links from them. If a page on the Internet has a link to a non-existent page on your site, then zbot will try to fetch that page from your site.
-
What IP address does zbot crawl from?
zbot is run on various machines that can change IP addresses over time. If you need to block zbot from accessing your site, add the following lines to your /robots.txt file:
User-agent: zbot
Disallow: / -
I have a question or comment that is not covered in this FAQ, who can I contact?
Please send mail to crawler at zvents dot com