One of our clients needed to advertise 15 feeds received from different sources and in different formats.
The client had an in-house full-time developer who tried to automate the feeds processing and merging to a unified format.
Also, values within fields category, location, country, and state needed to be normalized to match values on the job board.
Every time a new unknown value appears in any feed, it took plenty of time for the developer to make an appropriate change in processing scripts.
During one regular meeting, board executives decided to cut the costs. Their technical officer found 6 companies who do scraping and sent a test request to them. Only our company was able to provide a free demo to the client within 24 hours after the request. There were 3 test feeds processed and merged to the requested format and values normalized.
According to the client, he reduced his costs up to 10 times!
Which funds do you spend on data transfer? Ask Aspen Tech Labs for a free audit.
We noticed you mentioned scraping Indeed.com
Just to confirm: Indeed.com prohibits spidering of its content and they will block anyone trying to scrape it.
Normally, our clients ask us to spider jobs from direct employer websites and ATSes.
In some cases we can spider commercial job boards: if there is a formal agreement between our client and the job board to allow spidering.