For high quality data processing our WebSpiderMount system analyzes symbols of content (description f.e.) and determines any language with the help of the language detection tool. As a result, the system takes relevant mapping, dictionaries, indicates country, etc. in order to deliver tailored data to a client. All languages can be detected.
F.e. WebSpiderMount easily takes a certain/required language(s) (it can be one or several) content from any multi-language platform and pulls to client’s site. For multi-language sites our language detection tool scrapes and then pushes relevant language content to the site version.
We noticed you mentioned scraping Indeed.com
Just to confirm: Indeed.com prohibits spidering of its content and they will block anyone trying to scrape it.
Normally, our clients ask us to spider jobs from direct employer websites and ATSes.
In some cases we can spider commercial job boards: if there is a formal agreement between our client and the job board to allow spidering.