Process

Automated vs. Manual Accessibility Testing: When to Use Each

By EZUD Published · Updated

Automated vs. Manual Accessibility Testing: When to Use Each

Neither automated nor manual testing alone provides sufficient accessibility coverage. Automated tools are fast, scalable, and consistent. Manual testing is nuanced, contextual, and capable of evaluating subjective quality. A mature accessibility testing strategy uses both in complementary roles.

What Automated Testing Catches

Automated tools like axe-core (Deque), Pa11y, WAVE (WebAIM), and Accessibility Insights (Microsoft) evaluate rendered DOM and CSS against programmatic rules derived from WCAG 2.2 success criteria. They reliably detect:

  • Missing alt attributes on images
  • Insufficient color contrast ratios
  • Missing form labels or misassociated labels
  • Incorrect heading hierarchy (skipped levels)
  • Missing document language attribute
  • Duplicate element IDs
  • Missing landmark regions
  • ARIA attribute misuse (invalid roles, missing required attributes)

Deque’s research indicates that automated tools catch approximately 30-40% of WCAG 2.2 success criteria violations. This is a significant portion of the total issue landscape and includes many of the most common defects.

Strengths

  • Speed. A full-page scan takes seconds. CI/CD integration catches issues on every build.
  • Consistency. The same rules are applied every time. No variation between testers.
  • Scale. Hundreds of pages can be scanned in a single run.
  • Low cost. Many tools are open source (see open-source accessibility tools).

Limitations

  • Cannot evaluate whether alt text is meaningful or accurate.
  • Cannot determine whether focus order is logical.
  • Cannot assess whether a custom widget is operable in practice.
  • Cannot verify that screen reader announcements make contextual sense.
  • Cannot detect missing content (an image with no alt text is detectable; a missing image that should have been included is not).

What Manual Testing Catches

Manual testing involves a human evaluator using assistive technology, keyboard navigation, and expert judgment to assess the product against WCAG criteria and real-world usability.

Manual testing reliably detects:

  • Illogical focus order that disrupts task flow
  • Screen reader announcements that are technically correct but confusing in context
  • Keyboard traps in custom widgets
  • Missing or unhelpful error messaging for assistive technology users
  • Dynamic content updates that are not communicated to screen readers
  • Touch target issues on mobile devices
  • Reflow problems at high zoom levels
  • Cognitive complexity that impedes task completion

Strengths

  • Depth. Evaluators understand context, intent, and user experience.
  • Flexibility. Evaluators can explore unexpected paths and edge cases.
  • Completeness. Manual testing can address all WCAG success criteria, including those that require human judgment.

Limitations

  • Slow. A thorough manual audit of a single page can take 30-60 minutes.
  • Variable. Results depend on the evaluator’s expertise, experience, and assistive technology proficiency.
  • Expensive. Skilled accessibility auditors command premium rates.
  • Not scalable for every build. Manual testing cannot run on every commit.

The Complementary Model

The most effective approach layers automated and manual testing:

In development (every build)

  • Automated linting (eslint-plugin-jsx-a11y) catches issues in the editor.
  • Automated unit tests (jest-axe) verify component accessibility.
  • Automated CI/CD scans (axe-core, Pa11y) gate deployments.

In QA (every sprint or release)

  • Manual keyboard testing of new and changed features.
  • Screen reader testing with NVDA, JAWS, or VoiceOver for interactive components.
  • Manual review of dynamic content, focus management, and error handling.

Periodically (quarterly or after major changes)

Continuously

  • Automated monitoring dashboards track accessibility health over time (see continuous accessibility monitoring).
  • User-reported issues supplement automated and manual findings.

Choosing Tools

For automated testing, see our guide on open-source accessibility tools and resources. Key considerations:

  • axe-core for the lowest false-positive rate and broadest integration ecosystem.
  • Pa11y for CI/CD pipeline integration and scheduled monitoring.
  • WAVE for visual debugging during development.
  • Accessibility Insights for guided manual assessment workflows that bridge automated and manual testing.

For manual testing, ensure evaluators are proficient with the assistive technologies your users rely on. Refer to training developers on accessibility for building these skills internally.

Reporting and Tracking

Both automated and manual findings should feed into the same bug triage system. Track:

  • Source (automated scan, manual audit, user testing, user report)
  • WCAG criterion
  • Severity
  • Status (open, in progress, resolved, verified)
  • Time to resolution

Report through accessibility metrics dashboards to maintain organizational visibility into testing coverage and issue trends.

Key Takeaways

  • Automated testing catches 30-40% of WCAG issues and excels at speed, consistency, and scale.
  • Manual testing catches the remaining issues that require human judgment: focus order, screen reader experience, and usability.
  • Layer both: automated in CI/CD, manual in QA and periodic audits, user testing periodically.
  • Feed all findings into a single triage system regardless of source.
  • Neither method alone is sufficient; both are necessary.

Sources

More in Process

Inclusive Design Process Guide: Building Accessibility Into Every Stage Guide Continuous Accessibility Monitoring Accessibility in Design Reviews Nothing About Us Without Us: The Principle That Shapes Inclusive Design

View all Process articles →