Keep your job listings up-to-date with source Employer websites by utilizing Synchronization option of Job Wrapping. Sync option will make sure only new jobs are added to your system and expire the vacancies removed from source websites.
Job wrapping synchronization feature provides incremental downloads to make sure you don’t overload source websites with excessive requests, but only download new jobs added to system.
Job Sync requirements:
Job Sync feature requires HTTP/XML posting interface on a target job board.The interface is to provide “Add” and “Remove” commands. It is also to be able to return job ID to spider upon successful posting.
We noticed you mentioned scraping Indeed.com
Just to confirm: Indeed.com prohibits spidering of its content and they will block anyone trying to scrape it.
Normally, our clients ask us to spider jobs from direct employer websites and ATSes.
In some cases we can spider commercial job boards: if there is a formal agreement between our client and the job board to allow spidering.