WebSpiderMount introduces a new audit service for all existing scrapes in addition to automatic daily monitoring. Each audit is performed manually by our team to ensure that every job is captured and there are no issues with the scrape. This is an extra quality check by our team.
Key points of the auditing service:
-We are reviewing each scrape closely every 6 months, even if there are no errors/warnings reported by our monitoring system;
-Each scrape has 14 checklist points that are verified by the team. Some of the points are:
– We analyze audit results to further understand what issues are not yet covered by automatic monitoring system to improve it further
– We are now actively reviewing and verifying live scrapes, and you will be notified of the progress.
-Our customers will be notified via the Dashboard on the status of each scrape. Statuses such as: being audited at the moment, recently audited and pending audit will appear for each scrape. We will also notify our clients via email with some additional information related to scrape audit.
For more information, contact us here.
We noticed you mentioned scraping Indeed.com
Just to confirm: Indeed.com prohibits spidering of its content and they will block anyone trying to scrape it.
Normally, our clients ask us to spider jobs from direct employer websites and ATSes.
In some cases we can spider commercial job boards: if there is a formal agreement between our client and the job board to allow spidering.