Screen scraping: The basics

2009-09-30

Many may not be aware of screen scraping - its potential problems, risks and even who is taking the data in the first place. So what are the basics of this kind of data appropriation? And what kind of firms are or have been affected by screen scraping?

Anders Söderström, CEO of Sentor MSS, pointed out that screen scraping is a concern for many different forms of businesses. How it impacts organisations is dependent on what kind of company it is, he commented. Mr Söderström explained that with airlines, online travel agents scrape the data and make a decent amount of money from the information on hotels and rental cars that they take.

Airlines will end up losing out from being screen scraped as they will lose revenue from auxiliary services such as hotels and rental cars and other products on the website, he continued. This is particularly important for those low-cost carriers that make very little money on the actual flights, the expert noted.

As for other firms who are typically targets for screen scraping, data from business to business directories are scraped by competitors who take information from the listings from their own site, Mr Söderström pointed out. He said that the scrapers are looking to make their own databases more complete. "It is more interesting to go to a portal where you have as many products as possible," he continued.

Yellow pages are also frequently scraped by telemarketing companies, according to Mr Söderström. They may lose customer confidence as a result of this screen scraping because clients pay for a listing in a yellow pages and not to be contacted by telemarketers, he said, and business could also be lost to niche directories.

"For instance, if someone were to start up plumbers.com and manage to make that the primary source of contact information for plumbers, that would take a significant share of the traffic from the yellow pages," Mr Söderström added.

So what can Sentor do to help? Well, the expert noted that the company offers a variety of services. This includes consultancy around preventative measures, bespoke hardware and software solutions as well as a 24/7 managed anti scraping service.

Another service that the organisation can offer businesses is a Scraping Risk Assessment. With this organisations can learn quickly whether or not they are at danger of malicious scraping activity.

As for what this involves, Sentor loads a period of log data from the client's web-servers into its automated anti scraping surveillance network (ASSASSIN) system. This information is then is replayed live in ASSASSIN, allowing operators to analyse scraping events as if it was in real-time.

During this process, the client can access Sentor's Security Management Portal where they are able to download reports about scraping activity as well as general traffic at any time. Companies can monitor the progress of their Scraping Risk Assessment with this. Firms can also look at detailed information about each scraping incident that occurred on their site during the test period.

Once this is completed, a thorough report is provided, which also shows up other abusive behaviour. So perhaps it is now the time for many firms, particularly those that Mr Söderström said were more likely to be affected, for just such a test. As the saying goes, it is better to be safe than sorry.