Job wrapping or job scraping is the process of copying jobs from employer sites or ATSes, converting these jobs to the required format and posting resulting vacancies to a job board database.
Job spider technology downloads (scrapes) all or selected jobs, extracts job content from vacancy pages’ HTML source code, cleans up and improves job content, converts job data into an XML or other job board friendly file format, and synchronizes automatically via recipient job board API.
Step 1. Spider runs job search on employer website and browses job listings
Spider scrapes all jobs or uses search criteria, i.e. USA location and specific industry category only
Daily or more frequent spidering runs are scheduled for specific time(s) of day and days of wee
Resulting job listings are filtered out by keywords in job titles, description, etc.
Jobs are synchronized: spider downloads new ones, tracks updates and removes expired jobs
Job search form and results listing example:
Step 2. HTML source code is downloaded for each posting and job data is extracted
Job data content is parsed and converted into plain text or saved as HTML
Job description HTML tags are cleaned up or replaced, non-desired keywords (i.e. recruiter contacts) are removed
Parsed locations and categories data is mapped to recipient job board database listings
Job taxonomy is used to identify missing industry categories
Just to confirm: Indeed.com prohibits spidering of its content and they will block anyone trying to scrape it.
Normally, our clients ask us to spider jobs from direct employer websites and ATSes.
In some cases we can spider commercial job boards: if there is a formal agreement between our client and the job board to allow spidering.
This website stores cookies on your computer. These cookies are used to collect information about how you interact with our website and allow us to remember you. We use this information in order to improve and customize your browsing experience and for analytics and metrics about our visitors both on this website and other media.I agree and accept