Is ASSASSIN the answer for screen scraping battle?

2010-01-27

There are other ways besides to blocking screen scraping other than implementing Sentor's ASSASSIN (automated anti scraping surveillance network) software. But how useful are these other methods and what are they?

Well, one of them is captcha, which will be familiar among many web users. When people encounter captcha, they are presented with text and numbers, which they are then asked to type in to prove they are not a machine. There are a number of vulnerabilities with this system, however.

Fortunately for those wanting to use it, there have been a number of developments in captcha recently that could be beneficial for its effectiveness as a tool against screen scrapers.

A research project at Tel Aviv University is looking into the effectiveness of a video captcha code that may prove more difficult to get around. The technique generates images of 3D objects, such as a running man or a flying airplane. Security developers will be able to create as many moving "emergence" images as they like that computer algorithms will be unable to decode, explained Professor Danny Cohen-Or of Tel Aviv University's Blavatnik School of Computer Sciences.

The researchers defined "emergence" as the unique human ability to collect separate pieces of apparently useless information and then bring it together to see it as an identifiable whole. Computers do not yet have this capability.

Professor Cohen-Or explained the research in his own terms: "Humans have a very special skill that computer bots have not yet been able to master.

"We can see what's called an 'emergence image' - an object on a computer screen that becomes recognizable only when it's moving - and identify this image in a matter of seconds. While a person can't 'see' the image as a stationary object on a mottled background, it becomes part of our gestalt as it moves, allowing us to recognize and process it."

Despite this development, Sentor has pointed out a number of problems with captcha as a means of preventing screen scraping, one of which is that it can be annoying for users. Another issue is that people could simply get through captcha tests manually before carrying out screen scraping on a website.

Another method of stopping screen scraping is to blacklist IPs known to carry out such activities. However, there are flaws with this also. Sentor has noted that companies can end up blocking legitimate users with this method.

Amazon Web Services has recently hit problems in this area, but with email blacklisting. The company often has its IP addresses blocked, thereby stopping its users from sending out emails, SearchCloudComputing.com has reported.

"Certain spam filtering organisations require changes in our reverse DNS [Domain Name System] information in order to white list user's Elastic IPs. The work we are doing is aimed at making this easy for Amazon EC2 users," Amazon spokeswoman Kay Kinton told the website.

The firm revealed it is now testing changes to its DNS record system that will prevent email blacklisting for EC2 users.

So it seems that these other two methods have their flaws. But what are the benefits of using the Sentor ASSASSIN service? Firstly, the system is able to block sophisticated scraping attacks within minutes of them being detected.

To identify screen scraping activity, ASSASSIN analyses traffic and usage patterns on websites, raising the alarm when it detects suspicious scraping activity. On top of this, legitimate users will not be affected in any way, as Sentor's security analysts evaluate the scraping activity. Users can also access scraping incident reports constantly through Sentor’s online customer portal.

With all these benefits, and with the flaws of other methods, ASSASSIN could be the answer for those companies looking to block screen scraping activity.