Screen scraping target: Directories

2009-10-22

Directories can be negatively impacted by screen scraping in a number of ways and those involved in the industry may want to get some better protection than they already have in place.

While the paper product is still important in this sector, the rise of internet adoption means that some firms could be negatively impacted by screen scraping activity. They need to make sure that their directories are attractive if they want to remain competitive, according to Sentor. Directory companies should therefore look to protect their database from scraping and data theft.

Information is scraped from these directories for a number of reasons, the company notes. Firstly, scrapers will look to use what they find in a company's database to add to their own service. This means they will be able to offer the same value to both visitors and advertisers as the scraped company can, without having invested either the time or money. The scrapers will even be able to offer more than the affected party if they already have their own information in place.

Fraudulent directories, which scrape data from legitimate services and send out fake invoices or sell worthless ads on their website, can bring down the reputation of those in the industry, the Scandinavian IT firm points out. They may also end up taking traffic away from these real directory websites.

Back in 2006, Yell.com teamed up with Sentor to become the first firm to implement the latter's ASSASSIN (Automated Assessment Anti Scraping Surveillance Network) to help in counteract data scraping.

Nigel Ridgeon, head of analysis and information at Yell.com, explained why the directory company was adding this protection against scraping activity: "Yell.com is available for users wanting to make online searches for a product or for service suppliers. However, we do not licence the data for commercial or users' personal gain."

It is these kinds of misuses, such as people downloading data for sales prospecting, that led Yell.com to step up its moves to counter data scraping, Mr Ridgeon commented.

He asserted that the firm needed to make sure its detection and blocking systems were as sophisticated as they could be, adding that the Sentor solution offered Yell.com constant automated protection that is invisible to all its legitimate users.

Martin Zetterlund, senior consultant at Sentor, said at the time of the deal that the global IT security specialist had identified scraping as a rapidly growing threat to internet-based companies, such as those in the online directories, travel and dating spheres.

"So when Yell.com asked us to find a solution to its problem we were more than happy to develop ASSASSIN and incorporate the service into our portfolio," Mr Zetterlund commented.

Sentor's ASSASSIN service analyses Yell.com usage patterns in real time, detecting any suspicious behaviour. It then creates a scraping incident case, which is investigated by operators at Sentor's Security Operations Centre in accordance with the security company's client-specific scraping response process. If the activity is found to be the work of a scraper, then the operator issues a block order against that specific IP address, putting a stop to the data stealing immediately.

Perhaps this is something that other directories now need to consider. Even though the difficult economic times seem to be coming to a close, scrapers can and most likely will still attack.