Anyone who has worked on UI automation has been burned by selectors.
Change a button's label, and the test fails. Wrap the DOM in an extra layer, and the test fails. A designer tweaks a modal's position, and the test fails. Switch the rendering method in a mobile WebView, and the test still fails. Eventually, you realize that half the time spent on "automated testing" is actually just maintaining the tests themselves.
This is exactly why web-infra-dev/midscene deserves attention. It's not just another Playwright wrapper; instead, it shifts the entry point of UI automation from "finding a specific CSS selector" to "describing what I want to do." The repository description is straightforward: AI-powered, vision-driven UI automation for every platform. At the time of this snapshot, it had around 13,337 stars and 1,004 forks, gaining roughly 99 stars in a single day on GitHub's TypeScript Trending list.
The feature breakdown in the README is equally clear: you can describe your goals and steps in natural language, and Midscene will plan and operate the UI. Scripts can be written using the JavaScript SDK or YAML. For web scenarios, it integrates with Puppeteer or Playwright, or uses Bridge Mode to control desktop browsers. On mobile, it supports controlling Android via adb and iOS devices/simulators via WebDriverAgent.
The key here isn't "eliminating code entirely," but rather writing code in a more stable place.
Traditional UI automation focuses on elements: #submit-button, .modal .confirm, data-testid=save. Midscene focuses more on intent: click "Save", confirm the modal, check if an error message appears on the page. For frequently changing interfaces, this approach aligns much closer to human testing steps and serves as a far better operational interface for Agents.
Midscene also provides three categories of APIs: the Interaction API for manipulating the UI; the Data Extraction API for pulling data from the interface and DOM; and the Utility API, which includes helper functions like aiAssert(), aiLocate(), and aiWaitFor(). Additionally, it offers an MCP service that exposes Midscene Agent's atomic actions as MCP tools, allowing higher-level Agents to inspect and operate the UI using natural language.
This fits perfectly into a practical workflow:
Layer 1: Keep deterministic tests like Playwright to cover core paths. They are fast, stable, and ideal for CI.
Layer 2: Use Midscene to write smoke tests for "highly volatile interfaces," such as admin dashboards, operational configuration pages, and campaign landing pages. The DOM in these areas changes frequently, but the human verification steps remain relatively fixed.
Layer 3: Connect the Midscene MCP to a coding agent. After a frontend developer finishes modifying a page, the Agent can automatically open it, execute actions like "log in, navigate to the orders page, filter by status, and verify that data appears in the list," and then decide whether further fixes are needed based on screenshots and assertion results.
Of course, vision-driven automation is no silver bullet. It relies more heavily on model stability and can be slower than pure selector-based tests. For high-risk workflows like payments, permissions, or data deletion, you should still rely on more deterministic automation scripts and manual reviews as a safety net.
But the trend is already clear: UI automation is shifting from "machines reading the DOM" to "machines understanding the interface." Selectors won't disappear, but they shouldn't bear the entire cognitive load anymore.
If you want to trial this in your team, start with the most tedious but lowest-risk workflows: for example, backend filtering, form entry, or operational configuration previews. These scenarios change often, require frequent manual verification, and have controllable fallout if something breaks, making them ideal for visual automation to fill the gaps. Once stable, consider integrating some scripts into CI or handing them off to higher-level Agents. Don't jump straight into high-risk pipelines like payments, deletions, or permissions.
Primary Source: GitHub - web-infra-dev/midscene