How to Benchmark a DAST Scanner with OWASP Juice Shop

Before you run a DAST scanner against your production web application, you need to know what it can — and cannot — find. The fastest way to get that answer is to run it against OWASP Juice Shop: a deliberately vulnerable web application with a known, documented set of security flaws spanning every major category.

If your DAST scanner cannot detect the straightforward vulnerabilities in Juice Shop — including the unobfuscated SQL injection in the login form — it is not ready for your production application.

This guide covers the complete benchmark process: setting up a local Juice Shop instance, configuring authenticated scanning, recording which vulnerabilities the scanner should find at minimum, and applying a practical scoring rubric to compare DAST tools objectively.

Why Juice Shop Is the Standard Benchmark

OWASP Juice Shop was explicitly designed to test both human penetration testers and automated security tools. It contains:

Known vulnerabilities with documented categories — every flaw is categorized, scored by difficulty, and described
Modern architecture — Angular SPA frontend, Node.js REST API, JWT authentication, SQLite database — matching real-world web application patterns
Multiple vulnerability classes — SQL injection, XSS, SSRF, IDOR, JWT attacks, security misconfigurations, missing headers, and more
Both easy and hard variants — some vulnerabilities are deliberately obvious (for scanner benchmarking); others require human reasoning

The combination makes Juice Shop both a minimum bar and a ceiling test. A scanner that fails the obvious SQL injection is not viable. A scanner that also catches the JWT algorithm confusion vulnerability — which requires understanding token structure, not just sending payloads — is demonstrably more capable.

Step 1: Set Up a Local Juice Shop Instance

Never benchmark a DAST scanner against the public OWASP demo server. The public instance is shared, rate-limited, and prohibits automated scanning. Always use a local or private instance.

# Start Juice Shop with Docker (fastest method)
docker run -d -p 3000:3000 --name juice-shop bkimminich/juice-shop

Wait approximately 10 seconds for startup, then verify:

curl -I http://localhost:3000/
# Expected: HTTP/1.1 200 OK

Open http://localhost:3000/ in your browser. You should see a juice store with products, a navigation bar, and a login page.

For Scanner Benchmarking: Use a Stable Docker Image

If you’re comparing multiple scanners across several days, pin the Juice Shop version so you’re comparing against an identical attack surface:

# Use a specific tagged version for consistency
docker run -d -p 3000:3000 --name juice-shop bkimminich/juice-shop:v17.0.0

Reset Juice Shop Between Scanner Tests

To ensure each scanner starts from a clean state:

docker stop juice-shop && docker rm juice-shop
docker run -d -p 3000:3000 --name juice-shop bkimminich/juice-shop

Step 2: Create Test Accounts

Juice Shop has authenticated attack surface that’s invisible to unauthenticated scans. Before benchmarking, create test accounts:

Regular user:

Navigate to http://localhost:3000/#/register
Register with: [email protected] / Test1234! / security question answer: test

Admin account: The admin account email is revealed in one of Juice Shop’s challenges, but for scanner benchmarking, use:

Email: [email protected]
Password: This must be found through the challenges — alternatively, most scanner benchmarks focus on the authenticated regular user

For a comprehensive scanner test, create both a regular user and locate the admin credentials (via the login bypass SQL injection or through the challenge hints), then configure two separate authenticated scan sessions.

Step 3: Configure Authenticated Scanning

This is where most DAST scanner benchmarks fail: running only an unauthenticated scan and wondering why the results look shallow.

An unauthenticated scan of Juice Shop covers approximately 20% of its attack surface. The REST API (which contains IDOR, authentication-bypass, and authorization flaws) is entirely behind a JWT authentication layer. Without authentication, the scanner sees: the product listing page, the login form, the registration form, and a handful of public endpoints.

What to Configure for Authenticated Scanning

Login URL: http://localhost:3000/rest/user/login
Login method: POST

Login body:

{
  "email": "[email protected]",
  "password": "Test1234!"
}

Auth token extraction: Extract the authentication.token value from the JSON response
Token injection: Send as Authorization: Bearer <token> header on subsequent requests
Session validation: Check that authenticated pages (e.g., /#/profile) are accessible after login
Re-authentication: Configure re-auth when the token expires (Juice Shop tokens expire after some time)

Verify Authentication Is Working

After configuring authentication, manually verify the scanner is logging in correctly before starting a full scan. Check the scanner’s request log — you should see:

A POST to /rest/user/login with the credentials
Responses containing the JWT token
Subsequent requests with Authorization: Bearer <token> headers
200 responses (not 401) from authenticated endpoints like /api/BasketItems

If any of these are missing, the scanner is running unauthenticated and the benchmark will be invalid.

Step 4: Define the Minimum Expected Findings

A production-ready DAST scanner should find all of the following in Juice Shop on an authenticated scan. These are the minimum expected findings — they represent well-known, clearly exploitable vulnerability classes with obvious signals:

Critical / High Severity

Vulnerability	Location	How a Scanner Should Detect It
SQL Injection (Login)	`POST /rest/user/login` → `email` parameter	Error-based: `' OR '1'='1'--` triggers authentication bypass or SQL error
Reflected XSS (Search)	`GET /rest/products/search?q=`	`<script>alert(1)</script>` or `"><img src=x onerror=alert(1)>` reflected in response
Sensitive File Exposure	`GET /ftp/acquisitions.md`	Direct access to file that should be restricted
Directory Listing	`GET /ftp/`	Server returns a directory listing

Medium Severity

Vulnerability	Location	How a Scanner Should Detect It
Missing Content-Security-Policy	All pages	Absence of `Content-Security-Policy` response header
Missing X-Content-Type-Options	All pages	Absence of `X-Content-Type-Options: nosniff` header
Missing X-Frame-Options	All pages	Absence of `X-Frame-Options` header
CORS Misconfiguration	API endpoints	Test `Origin: https://evil.com`; check if reflected in `Access-Control-Allow-Origin`
Cookie Missing HttpOnly	Login response	`Set-Cookie` for session without `HttpOnly` flag
Cookie Missing Secure Flag	Login response	`Set-Cookie` without `Secure` flag

Scanner-Differentiating Findings

These vulnerabilities separate adequate scanners from excellent ones:

Vulnerability	Location	Difficulty for Scanners
Reflected XSS (Order Tracking)	`GET /track-result?id=`	Input reflected in page; requires tracking less obvious parameter
SSRF via image URL	Profile image URL parameter	Requires SSRF payload handling, out-of-band detection
Insecure JWT (missing validation)	All API endpoints	Requires understanding JWT structure and forging
Stored XSS via product reviews	POST to reviews, then GET product page	Requires full crawl and stored XSS detection across requests
IDOR on basket API	`GET /api/BasketItems/:id`	Replace `:id` with another user’s basket ID; requires authentication

Step 5: Run the Benchmark

Structure your benchmark as two runs per scanner:

Run 1: Unauthenticated Scan

Configure the scanner with no credentials and scan http://localhost:3000/. Record:

Total findings count
Critical/High/Medium findings
Which expected findings from the table above were detected
Crawl coverage (how many unique URLs discovered)
Scan duration

Run 2: Authenticated Scan

Configure authentication as described in Step 3 and scan again. Record the same metrics plus:

Whether authenticated pages were reached (check crawl log for /#/profile, /api/users/me, /api/BasketItems)
IDOR findings on basket/order endpoints
JWT-related findings
Findings that only appear in authenticated areas

The ratio of authenticated to unauthenticated findings reveals how deeply the scanner understands authenticated web application scanning. A scanner that finds the same number of issues in both runs is not successfully authenticating.

Step 6: Scoring Rubric

Use this rubric to compare DAST scanners objectively:

Mandatory Pass / Fail (Minimum Bar)

A scanner that fails any of these is not viable for production use:

SQL injection in login form detected — the most obvious SQLi test case
Reflected XSS in search detected — basic reflected XSS with trivial payload
At least 3 of 6 security header findings detected — deterministic header checks
Authentication succeeds — authenticated pages are reached in the authenticated scan

Scored Criteria (0–10 per category)

1. Injection Detection (0–10)

0–3: Misses the login SQLi
4–6: Finds login SQLi and search XSS, misses the order tracking XSS and blind injection
7–9: Finds all obvious injection points; detects some parameter contexts others miss
10: Finds all injection points including stored XSS via reviews and blind injection

2. Authentication and Session Analysis (0–10)

0–3: Cannot authenticate or finds nothing behind authentication
4–6: Authenticates, finds basic authenticated findings
7–9: Finds IDOR on basket API, detects session security issues
10: Finds IDOR, JWT security issues, and session management weaknesses

3. Security Header Coverage (0–10)

0–3: Finds fewer than 3 missing headers
4–6: Finds 3–5 missing headers
7–10: Finds all missing headers with correct severity assignment

4. Crawl Coverage (0–10)

Measure: unique endpoints discovered in authenticated scan
0–3: Fewer than 20 unique endpoints (shallow crawl)
4–6: 20–50 unique endpoints
7–9: 50–100 unique endpoints
10: 100+ unique endpoints, including REST API paths

5. False Positive Rate (0–10)

Count findings that are clearly not vulnerabilities (e.g., false SQL injection in search params that are actually safe)
0–3: More than 30% false positive rate
4–6: 10–30% false positive rate
7–9: 5–10% false positive rate
10: Under 5% false positive rate with confirmed exploitable findings

Total: /50

Common Benchmark Mistakes

Running Against the Public Demo Server

The public Juice Shop demo at demo.owasp-juice.shop is shared, rate-limited, and prohibits automated scanning. Results will be meaningless and you will be interfering with other users. Always use a local instance.

Accepting the Unauthenticated Scan as the Complete Test

Many vendor proof-of-concept scans are run unauthenticated because it’s simpler to configure. An unauthenticated scan leaves 80% of Juice Shop’s attack surface untested. Insist on authenticated scanning in any evaluation.

Not Resetting Between Scanner Tests

If you run Scanner A first and it leaves a SQL error state in Juice Shop’s database, Scanner B may get different responses. Always reset to a fresh Docker container between scanner runs.

Not Reviewing False Positives

A scanner that reports 200 findings in Juice Shop sounds impressive until 150 of them are false positives. Count confirmed findings (where the scanner provides a request/response proving the finding is real) separately from potential findings.

Ignoring Scan Duration

Scan speed matters for CI/CD integration. A scanner that takes 8 hours to scan Juice Shop will not work in a PR-level CI/CD gate. Record scan duration for both unauthenticated and authenticated runs.

Interpreting Your Results

Interpretation: The scanner meets the minimum bar. It detects obvious, well-documented vulnerabilities. Suitable for low-risk internal applications where the primary value is catching regressions.

Limitations: Complex injection patterns, stored XSS, JWT vulnerabilities, and IDOR require more. Verify that what it finds in Juice Shop it can also find in your actual application stack.

Scanner Finds All Mandatory + Some Scored Items

Interpretation: A capable production DAST tool. Proceed to test against a staging version of your actual application.

Next step: Focus on the categories where the scanner scores low in Juice Shop — those categories are likely to be the blind spots in your real application scan.

Scanner Finds Everything Including IDOR, JWT, Stored XSS

Interpretation: A high-performing scanner. Proceed to evaluate deployment model (SaaS vs. on-premise), CI/CD integration quality, false positive rate at scale, and pricing.

What Offensive360 DAST Finds in Juice Shop

Offensive360’s DAST scanner is benchmarked against Juice Shop on every release as part of the continuous testing pipeline. On a full authenticated scan:

Detected findings include:

SQL injection (login form, search endpoint)
Reflected XSS (search bar, order tracking)
Stored XSS (product reviews — requires authenticated crawl and multi-request correlation)
Missing Content-Security-Policy, X-Content-Type-Options, X-Frame-Options
Directory listing on /ftp/
Sensitive file exposure (/ftp/acquisitions.md)
CORS misconfiguration (reflected origin header)
Insecure cookie configuration (missing HttpOnly, missing Secure flag)
IDOR on basket API (requires authenticated scan with two user accounts)
JWT token not validated on unsigned tokens (requires JWT-aware testing)

The scanner is configured to re-authenticate automatically when JWT tokens expire during long scans, ensuring deep authenticated crawl coverage throughout the test.

To run Offensive360’s DAST scanner against OWASP Juice Shop or your own application: Book a demo — results in minutes, source code never leaves your network.

Frequently Asked Questions

Can I use Juice Shop to benchmark SAST tools too?

SAST tools analyze source code, not running applications — so the testing methodology is different. For SAST benchmarking, download the Juice Shop source code from github.com/juice-shop/juice-shop and run your SAST scanner against it. Note that Juice Shop’s Node.js/TypeScript architecture is complex — SAST tools with weaker JavaScript/TypeScript support will struggle. Look for detection of the SQL injection in the models/user.js file and the missing sanitization in the search service.

What version of Juice Shop should I use for benchmarking?

Pin to a specific version tag (e.g., bkimminich/juice-shop:v17.0.0) for reproducible benchmarks. The latest tag updates with each release, which can change the vulnerability count and the challenge set. Using a specific version ensures your benchmark results are comparable over time and between tools.

How should I handle the JWT vulnerabilities in Juice Shop?

JWT vulnerabilities in Juice Shop (algorithm confusion, unsigned token acceptance) require the DAST scanner to actively forge JWT tokens — not just passively observe them. Most DAST scanners do not do this. If evaluating a scanner for an application that uses JWTs, this is a critical gap to verify separately, as JWT vulnerabilities are one of the most common high-severity findings in modern API security assessments.

Is OWASP Juice Shop a good benchmark for API security scanners?

Yes. Juice Shop’s REST API at http://localhost:3000/api/ is documented at http://localhost:3000/api-docs (Swagger/OpenAPI format). Import the Swagger spec into your API security scanner and run a dedicated API scan. Compare findings against the manual Juice Shop scoreboard — a good API scanner should detect IDOR on basket and order endpoints, injection in API parameters, and missing authentication on some endpoints.

How long should a full Juice Shop benchmark scan take?

With a modern DAST scanner, an unauthenticated scan should complete in under 30 minutes. An authenticated scan with deep crawl typically takes 45–90 minutes. If a scanner takes more than 2 hours to scan Juice Shop (a small application), it will not be practical for CI/CD integration against production applications.

Summary

Benchmarking a DAST scanner against OWASP Juice Shop is the fastest way to evaluate scanner capability before committing to a subscription or deployment. The process:

Start a fresh local Juice Shop instance (docker run -d -p 3000:3000 bkimminich/juice-shop)
Create test user accounts for authenticated scanning
Configure authentication with JWT token extraction and injection
Run unauthenticated and authenticated scans — record findings separately
Score against the minimum expected findings list — SQL injection, XSS, headers, directory listing
Apply the scoring rubric for differentiated evaluation
Reset and repeat for each scanner being evaluated

A scanner that passes the Juice Shop minimum bar — and specifically one that succeeds at authenticated scanning — is ready for evaluation against your actual production application stack.

Offensive360 DAST is tested against OWASP Juice Shop on every release. Book a demo to see it in action against Juice Shop or your own application — authenticated scanning, results in minutes.