Friday, November 03, 2006

Comparison between Appscan vs Webinspect

Last month I got a chance to evaluate the two popular vulnerability assessment tools Webinspect and Appscan and I wanted to share my findings with others. As you will notice, currently I have published only few technical comparison I will add more to it sooner. This comparison is strictly between Appscan 6.5 and Webinspect 6.2. Both the companies have since come with a beta release of a new version of their products and this comparison may not be valid for their new versions.
The evaluation was done on a dynamic web application with approx 1800 web pages.

  • Scan duration: On a website of approx. 1800 pages, Appscan took around 6 hours whereas webinspect took more then 12 hours (i had to stop it there as it would have taken a lot more time). Then i removed post-data injection from the list of attacks and the entire scan was completed in approx 4 hours.

  • Ability to pause a scan: Both the tools give you an ability to pause a scan and restart later.

  • Save a partial scan and restart later: Both the tools allow you to save a partial scan and restart it later.

  • What if webapp stops responding during scan: If your webapp stops responding during the scan, Webinspect displays an error and pauses the scan, so you can fix the problem and resume the scan later. Though appscan does the same but since the pause button is not on the toolbar and is as a dropdown it get a little confusing.

  • Change the network in the middle of a scan session: While a scan is in progress, if you want to pause it and resume from your home over a vpn connection, Appscan lets you resume the scan without any problems whereas Webinspect gave me an error. Though I just tried it once, I am not too sure if this is a problem with Webinspect. You can check it on your own if you're going to need this feature.

  • Maximum thread that can run simultaneously: Appscan has maximum of 10 threads whereas Webinspect has 75. You can customize the number of threads you want to run simultaneously. However, if you use more then 15 threads in Webinspect, it becomes very resource intensive.

  • Pick and choose attacks for a scan session: Both Appscan and Webinspect lets you do that.

  • Skip an attack while the scan is in progress: Webinspect lets you skip an attack while the scan is in progress whereas Appscan doesn't.

  • View the actual attack during a scan session: Webinspect displays the actual attack string on the status bar during the scan and also if a vulnerability was found whereas in Appscan you can only view if there was a vulnerability found.

  • Customize the order of attack for a scan session: Neither of the product lets you customize the order of scan. So, if you want to run Cross site scripting and SQL injection first then you will have to create a new scan session and just opt for these two attacks and remove all others from the list.

  • Custom attack scripts: Both Appscan and Webinspect lets you create a custom attack script as a macro. Webinspect however lets you create a custom attack agent using VB script.

  • Vulnerability Database updated: Both the product update their vulnerabilities regularly.

  • Vulnerabilities: Appscan found more vulnerabilities with more variants in a scan as compared to Webinspect. One other difference between the two products in terms of vulnerabilities is if there are 200 pages with same vulnerabilities, Appscan will display 1 vulnerability but can drill down to all the 200 pages whereas Webinspect will display as 1. Having said that, Appscan still detects more types of vulnerabilities then Webinspect.

  • Login module at the start of scan:Appscan lets you record a login sequence as a macro whereas Webinspect gives you more options. It lets you record a login sequence as a macro or enter the authentication credentials in the tool itself. The problem however with entering the authentication credentials inside the tool is that if they are wrong, Webinspect does not detects them as wrong and goes ahead with the scan and hence may not scan the entire site. In the other scenario, if you are using macro to record the login sequence, then Webinspect records the complete login and logout sequence whereas Appscan only records the login sequence.

  • Bookmark of follow up flag on a specific attack during a scan session: None of the tools currently have this feature.

  • External tools: Both the products have additional tools bundled with them. Watchfire's power tools are not integrated with Appscan and can be downloaded and used separately. Appscan, however, provides a framework where external tools can be invoked from within Appscan. Webinspect on the other hand, have more tools and all of them are tightly integrated within the product. Webinspect does not allows you to invoke an external tool from within the application.

  • Appscan scans for certain infrastructure vulnerabilities (like Apache, IIS, etc) whereas Webinspect does not.


This is by no means a complete report. I will add more points to it sooner so please check back again. Also, if there is something additional you would like to see in this report or if you would like to add to this report or even make changes to this report, please email me at anurag.agarwal "at" yahoo.com. I also got a chance to look at the beta release of Appscan 7.0 and Webinspect phoenix release and if you are looking to purchase one of these two products then I would strongly recommend looking at their beta releases of new versions.

21 comments:

Anonymous said...

Thanks, i've been looking for a recent review of both products

Anonymous said...

Nice, thanks for posting this.

> Scan duration: On a website of approx. 1800 pages, Appscan took around 6 hours whereas webinspect took more then 12 hours

Is this normal scan duration? Are there metrics available as far as how long scans are expected to take? I know its largely is dependent on the structure of the website, but we shouldn't we be able to ball part it?

> What if webapp stops responding during scan: If your webapp stops responding during the scan, Webinspect displays an error and pauses the scan, so you can fix the problem and resume the scan later.

Does this imply the website crashed as a result of the scan?

>Change the network in the middle of a scan session: While a scan is in progress, if you want to pause it and resume from your home over a vpn connection, Appscan lets you resume the scan without any problems whereas Webinspect gave me an error.

Wow, I never would have thought that this would have been an issue to address. Nice work Watchfire.

> Vulnerabilities: Appscan found more vulnerabilities with more variants in a scan as compared to Webinspect.

Any way to tell what issues, if any, were missed?

Can either of the scanners re-test individually found vulnerabilities?

Anonymous said...

It is possible to pause WebInspect, disconnect from the network and continue the scan once you have connected at another network location. Some of the WebInspect Toolkit tools, can be launched from within WebInspect, e.g. HTTPEditor, SOAPeditor etc. Personally, I use the WebInspect toolkit more than the main product itself. I have found many bugs in the toolkit and SPI are good at fixing bugs in the toolkit fairly quickly. I have never used AppScan but tested their toolkit when it first came out and it was much lamer than the WebInspect toolkit. As they saying goes, a fool with a tool is still a fool. - The Fool

Anonymous said...

It is possible to pause WebInspect, disconnect from the network and continue the scan once you have connected at another network location. Some of the WebInspect Toolkit tools, can be launched from within WebInspect, e.g. HTTPEditor, SOAPeditor etc. Personally, I use the WebInspect toolkit more than the main product itself. I have found many bugs in the toolkit and SPI are good at fixing bugs in the toolkit fairly quickly. I have never used AppScan but tested their toolkit when it first came out and it was much lamer than the WebInspect toolkit. As they saying goes, a fool with a tool is still a fool. - The Fool

Anonymous said...

I think the comparison as you said is not complete enough. There are many other parameters you can compare if you want to get a better idea of the tool you will choose.

I’ll do myself those questions:

Does both tools provide third party vulnerability identifiers BID, CVE/CAN ?

Got CVSS score?.

Does the tool include self explaining evidences?.

Can you change vulnerability risk level.

How my false positive does it get?

Can you delete (before report generation) false positives?.

What about log detail level?.

Can you create policy compliant policies, SARBANES OXLEY, OWASP, etc?

Anyway great post.
Regards,

freed0m at hacktimes.com

Anonymous said...

Thanks for the analysis Anurag, some great stuff! I mainly use WebInspect so I will add a few updates to your posting from what I have found personally.

I think WebInspect has many more vulnerability checks, something like 5500, so that could be why you were seeing it take a bit longer. Be sure to check out the Policy Manager to find all of the infrastructure vulnerabilities that it looks for. Also, WebInspect's tools can all be launched from within WebInspect or the program group in the start menu. I agree with "The Fool" in the fact that I find myself using the tools quite often.

I have also seen a beta of the new version of WebInspect and it's quite nice ;)

Anonymous said...

Anurag,

Can you reveal what was the scanner you decided to go with, and why?

Thanks.

Anonymous said...

BTW - regarding the toolkits, Watchfire has improved the toolkit, but the new tools are only bundled with the product and are no longer free of charge.

Anurag Agarwal said...

At 2:33 PM, Anonymous said...
It is possible to pause WebInspect, disconnect from the network and continue the scan once you have connected at another network location. Some of the WebInspect Toolkit tools, can be launched from within WebInspect, e.g. HTTPEditor, SOAPeditor etc. Personally, I use the WebInspect toolkit more than the main product itself. I have found many bugs in the toolkit and SPI are good at fixing bugs in the toolkit fairly quickly. I have never used AppScan but tested their toolkit when it first came out and it was much lamer than the WebInspect toolkit. As they saying goes, a fool with a tool is still a fool. - The Fool


I can't speak for earlier products. I did a comparison of Appscan 6.5 and Webinspect 6.2. The tools are provided for power users and of course depending on the knowledge level of the person running the product, they can be a big help in identifying additional vulnerabilities. But the bigger question here is how many of the users using either of these products are actually using their tools as well. I can bet most of the people are just running the automated scan before deploying their application into production.


hacktimes.com said...
I think the comparison as you said is not complete enough. There are many other parameters you can compare if you want to get a better idea of the tool you will choose.



I totally agree with you.. and as i mentioned in the post that this is by no means a complete report and i will publish more as soon as i get to it

At 5:02 AM, Joe Yeager said...

I think WebInspect has many more vulnerability checks, something like 5500, so that could be why you were seeing it take a bit longer. Be sure to check out the Policy Manager to find all of the infrastructure vulnerabilities that it looks for. Also, WebInspect's tools can all be launched from within WebInspect or the program group in the start menu. I agree with "The Fool" in the fact that I find myself using the tools quite often.

I have also seen a beta of the new version of WebInspect and it's quite nice



Joe - thanks a lot for your comments and if you read my post then i did mention that when i removed formdata injection the actual time was a lot less then appscan. I have also seen phonenix beta release and i like it too but i am yet to do a comparison between appscan 7 and webinspect new release.

Anonymous said...

"as i mentioned in the post that this is by no means a complete report and i will publish more as soon as i get to it"

I'd be especially curious if you can post how similar the configs are, or whether you can get them close. You mentioned you could make a change to WebInspect to drop it from 12+ hrs to 4 hrs, but was AppScan configured similarly...?

Anurag Agarwal said...

I'd be especially curious if you can post how similar the configs are, or whether you can get them close. You mentioned you could make a change to WebInspect to drop it from 12+ hrs to 4 hrs, but was AppScan configured similarly...?

I scanned with the entire attacks possible in their database. Webinspect was doing all the possible formdata injection as well and i noticed that was taking too long. I removed formdata injection from that list of attacks and then the rest of the attacks were completed fairly quickly. Hope that helps

Anonymous said...

Joe Yeager next time you do a stealth company posting use a fake name at least jeeze.

http://portal.spidynamics.com/blogs/yeager/default.aspx

Of course you use webinspect, you work for SPI Dynamics!

Anonymous said...

I noticed there was no mention of accuracy when referring to vulnerabilities(false positives). In the past 6-9 months I have seen the dependability of WebInspect surpass AppScan by a great deal, especially in the areas of Cross Site Scripting and SQL Injection.

Anonymous said...

Hi John,

Vulnerability metrics are very interesting to me. I'm curious to know more about how you measure your products performance vs another. For instance what's your process like? What types of websites do you test on? How many? etc. Any insight is helpful.

Anurag Agarwal said...

John Terrill said
WebInspect surpass AppScan by a great deal, especially in the areas of Cross Site Scripting and SQL Injection

John - In my assessment, it was not the case. Appscan found multiple variants of XSS vulnerabilities whereas Webinspect did not even mention half of those variants. As for SQL injection, though Appscan found one but can't comment on webinspect as it crashed every time it started looking for SQL injection attack. i reported the bug to SPI dynamics.

Anonymous said...

How come you decided to disable formdata tampering in WebInspect? this actually means that WI did not perform all the possible checks like AppScan did.

Comparing scan performance, while one product doesn't check thousands of tests is inaccurate.

This brings me to the conclusion that AppScan had better performance in your benchmark?

Anurag Agarwal said...

How come you decided to disable formdata tampering in WebInspect? this actually means that WI did not perform all the possible checks like AppScan did.

I disabled formdata injection as webinspect was taking too long a time to test all the possible combinations.
Maybe appscan did not test all the possible combinations as Webinspect was trying to do. I cant be sure about this.

Anonymous said...

Pick and choose attacks for a scan session: Don't you think WebInspect gives more control to the user in terms of picking up which variants of the attack (e.g. XSS) can be selected/deselected. In AppScan i could only enable or disable XSS altogether.

AppScan found more attacks: From my experience i can only tell that this varies from application to application. Have you tested this on different applications?

Could you include another factor in your comparison. No.of pages crawled. I think this is where both the tools differ from each other. I've had problems with AppScan while crawling some weird applications but WebInspect's crawling engine is reliable but takes time (in few instances it took more than 24 hrs for me).

Anonymous said...

In response to this article, which I think is actually pretty good for the uninitiated, I have a few minor points to make.

Scan Duration:
Of course there are default settings that effect the time it takes to run a scan. Were any of these settings equalized to compare apples to apples?
If you test an application with one form and one page you will see that WebInspect actually does more, which takes longer and it generates fewer false positives and redundancy, which suggests more concise reporting.

Vulnerabilities:
What do you mean by "types" of vulnerabilities?
If I have bad tires on my car the salesman will tell me I will get a flat by hitting a nail or glass or rocks or debris. The point is, I have to fix the tire. If I fix it correctly, it will solve the problem. Isn't the more important point that I fix the tire? I don't need the excessive reporting.

Login Module:
On WebInspect you confused application authentication with server authentication. The server authentication credentials (Basic or NTLM) can be provided in the tool whereas webform authentication credentials can only be passed through the use of a login script.
If you record the "logout" sequence then you have done the recording incorrectly. You NEVER record the logout when you create the login script.

"Appscan scans for certain infrastructure vulnerabilities (like Apache, IIS, etc) whereas Webinspect does not."
This is purely inaccurate and not true. WebInspect has hundreds of server, webserver and app server level vulnerabilities.

Anonymous said...

what extra manual checks are required in using either of the tools.

Anurag Agarwal said...

This post is more then a year old and both the vendors have since released new versions which have several improvements and new features. I am not sure if this comparison would hold true now.

Maybe someone else can do another comparison with the latest versions in the market.