There is an awful lot of great data online which has the potential to power awesome next-generation apps and services. Open data, government data services, and many APIs provide relatively easy access to data which can be integrated with sufficient time and effort.
However, the data your app or service needs is not necessarily available in a useful format. In fact, the vast majority of online data is locked up in web pages with no provision of data dumps or APIs in order to programmatically access it.
Frequently, the solution to this problem has been the development of so-called "scrapers". While scrapers may initially solve the problem of data retrieval, they come with a number of issues. First of all, they are technically difficult and expensive to implement, and even more so to maintain. Websites where the layout changes frequently and subtly can cause maintainers headaches while they fix the problems, with no data access in the meantime. Additionally, scrapers are specificRead More »from Mixing Up Great Data Without the Hassle of Scraping