Job wrapping or job scraping is the process of copying jobs from employer sites or ATSes, converting these jobs to the required format and posting resulting vacancies to a job board database.
Job spider technology downloads (scrapes) all or selected jobs, extracts job content from vacancy pages’ HTML source code, cleans up and improves job content, converts job data into an XML or other job board friendly file format, and synchronizes automatically via recipient job board API.
Step 1. Spider runs job search on employer website and browses job listings
Spider scrapes all jobs or uses search criteria, i.e. USA location and specific industry category only
Daily or more frequent spidering runs are scheduled for specific time(s) of day and days of wee
Resulting job listings are filtered out by keywords in job titles, description, etc.
Jobs are synchronized: spider downloads new ones, tracks updates and removes expired jobs
Job search form and results listing example:
Step 2. HTML source code is downloaded for each posting and job data is extracted
Job data content is parsed and converted into plain text or saved as HTML
Job description HTML tags are cleaned up or replaced, non-desired keywords (i.e. recruiter contacts) are removed
Parsed locations and categories data is mapped to recipient job board database listings
Job taxonomy is used to identify missing industry categories
Job replication is used to create vacancies for additional locations
Job advert example:
HTML source example:
Step 3. Parsed jobs data is saved into XML or CSV file and posted to job board API
Recipient job board XML or CSV data format is mapped with the job data extracted
Jobs are synced with target job board(s): new jobs are added, content updates are made and expired jobs are removed
Jobs posting and API options are: HTTP, SOAP, FTP, Amazon S3 or via email (view all)
Just to confirm: Indeed.com prohibits spidering of its content and they will block anyone trying to scrape it.
Normally, our clients ask us to spider jobs from direct employer websites and ATSes.
In some cases we can spider commercial job boards: if there is a formal agreement between our client and the job board to allow spidering.
Download our Job Scraping White Paper
If you are new to the jobs industry or just starting your job board, you need to ask these 8 questions about job scraping.
Download our 2022 Job Market Pulse Snapshot
This report highlights a variety of insights that can be drawn from our database of more than 9 million vacancies worldwide from over 100,000 different companies, updated daily