Automating Local Web Testing with Catcher

Practical Summary and Prerequisites

Catcher is an open-source, local-first desktop application designed for web testing that allows users to describe test scenarios in natural language. Unlike cloud-based SaaS testing platforms, Catcher operates entirely on your local machine, ensuring that your session data, cookies, and screenshots remain private. It utilizes a Bring-Your-Own-Key (BYOK) model, requiring users to provide their own API keys for LLM providers such as OpenAI, Anthropic, or Gemini. To get started, you need a machine running Windows or macOS and a valid API key for an LLM that supports vision capabilities, which is essential for the tool's coordinate-based fallback mechanism.

Installation and Initial Configuration

You can download the latest installers from the official releases page. Note that the installers are currently unsigned, so you may need to bypass security warnings on Windows or macOS during the first launch. Once installed, the primary configuration occurs within the application's Settings menu. You must select an LLM provider and input your API key. The application stores these settings locally in ~/.catcher/settings.json. It is recommended to verify that your chosen model supports vision, as this is required for the tool to perform coordinate-based clicks when standard selector strategies fail.

Writing Effective Test Steps

Catcher interprets natural language instructions to drive a Playwright-powered browser. To ensure high reliability, follow the documented best practices for prompt writing. Use quoted literals for specific elements or text to trigger deterministic substring matches, which are faster and more accurate than relying solely on AI interpretation. For example, you should structure your test steps as follows:

Click the 'Sign in' button
Type 'alice@example.com' in the email field
Type 'hunter2' in the password field
Click the 'Continue' button
Verify the page contains 'Welcome, Alice'

Maintain a one-action-per-step approach to simplify the planning process for the LLM. When defining assertions, quote the exact text that a user would see on the rendered page.

The Execution Workflow

When you trigger a test, Catcher follows a structured execution pipeline. First, it captures a snapshot of the page, including the ARIA tree and ranked clickable elements. It then attempts a heuristic match to identify targets without invoking the LLM, which serves as a fast path for simple interactions. If the heuristic is not confident, the tool sends the snapshot and the step description to the LLM to generate a plan. The execution engine then attempts to perform the action using Playwright. If standard selectors fail, the tool employs a vision-coordinate fallback, where the LLM analyzes a screenshot to determine the precise {x, y} coordinates for the interaction.

Monitoring and Debugging

The application provides a live run drawer that streams the browser viewport and displays the reasoning behind each step. This allows you to observe exactly what the planner is doing in real-time. If a test fails or produces low-confidence results, the runner automatically attempts to re-snapshot and re-plan based on the configured retry limit. You can audit the communication between the application and your LLM provider by enabling the 'Log all LLM calls' option in the settings. This transparency is useful for debugging complex interactions or verifying that the tool is correctly interpreting your natural language instructions.

Known Limitations and Privacy

While Catcher offers significant advantages for local testing, there are documented limitations. The current installer is limited to Chromium, though other browsers may be supported in a development environment. Because the tool relies on vision-based fallback, the accuracy of these interactions is dependent on the capabilities of the specific LLM model you select. Furthermore, since the binaries are currently unsigned, users must manually authorize the application on their operating systems. Regarding privacy, Catcher does not implement telemetry or analytics, and all site data and run history are stored locally in the ~/.catcher/ directory.

When to Use Catcher

Catcher is best suited for developers and QA engineers who require a local-first testing solution that avoids the security and privacy concerns of cloud-based SaaS platforms. It is particularly effective for testing applications with dynamic content, overlays, or complex animations where traditional CSS selectors might be brittle. By combining heuristic matching with LLM-driven vision fallback, it provides a robust way to automate end-to-end tests using natural language. It is an ideal choice for teams that want to integrate AI-driven testing into their internal workflows while maintaining full control over their data and infrastructure.

Contributing and Development

For those interested in extending the tool, the project is open-source and welcomes contributions. The codebase is structured to separate the main Electron process from the React renderer, with core logic for heuristics and plan parsing located in the electron/ directory. Developers should run npm install and npm run dev to start the application in watch mode. Before submitting pull requests, ensure that your changes pass the unit tests located in electron/__tests__/ by running npm test. Detailed guidelines for contributing are available in the repository's CONTRIBUTING.md file.