翻訳待ち：First Orion accelerates QA automation using Amazon Nova Act

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

This post is co-written with Mark Himelfarb and Garrett Wilkerson from First Orion. First Orion’s engineering teams were shipping faster than quality assurance (QA) could test, until Amazon Nova Act transformed QA automation. As a branded communications company whose solutions reach hundreds of millions of phone calls across carriers in the US, Canada, UK, and Germany, First Orion needed their QA to keep pace with rapid development velocity. The solution was a shift from script-based test automation to agents driven by AI that understand web interfaces the way a human does. This post describes how First Orion adopted Amazon Nova Act, the architecture they built around it, and the results they achieved. First Orion was founded in 2008 with a single belief: every communication should be clear, trusted, and recognized. Today they have over 300 team members across offices in North Little Rock, Seattle, London, and Dubai. Their solutions run across all major US carriers including T-Mobile, Verizon, AT&T, and Boost Mobile, major Canadian carriers, Vodafone in the UK, and Deutsche Telekom in Germany. Coverage is increasing with their Global Exchange. They help businesses connect with customers through branded calling, messaging, and identity solutions while protecting both parties from spam, scam, and spoofing. They serve a wide range of customers, from small businesses to global enterprises, including Fortune 500 companies. Their product suite covers the full communications lifecycle: INFORM Branded Calling, ENRICH Branded Messaging, AFFIRM Number Monitoring, SENTRY Call Blocking, and PROTECT+ Risk Detection. They believe the future of communication is built on trust, transparency, and intelligent technology, and innovation is central to their strategy. They invest in AI, conversational intelligence, and data-driven platforms to continuously improve how brands engage with their customers. That growth created an engineering challenge. As First Orion expanded into the small and medium-sized business (SMB) market alongside their enterprise customers, their web applications multiplied, and so did the QA burden of testing them. Decentralized portals outpaced QA team First Orion moved from a monolithic web portal to a decentralized, product-specific, modular, cell-based architecture. In this model, each team owns the applications for its own line of business and synchronizes only over a small, shared piece of the underlying platform. This let teams move in parallel at higher velocity, though it required substantial re-engineering of the platform to support. At the same time, as First Orion pushed further into the SMB market, the range of device form-factors and browser versions their customers used grew significantly, multiplying the combinations every application had to be tested against. UI testing quickly became the bottleneck. QA teams could not keep up with the sprawling set of web applications, and the consequences compounded: release velocity stalled, new features shipped with lower quality, and engineering time was increasingly consumed by regressions rather than new capabilities. First Orion’s QA team includes both QA Analysts and QA Automation Engineers. Automation Engineers build durable test suites from test case definitions and other documentation, while QA Analysts focus on requirement analysis, test case definition, exploratory testing, and software-quality risk management. But the incumbent test automation frameworks they relied on were inherently fragile, and three problems in particular slowed the team down. First, regression testing was not self-service. Developers needed a straightforward way to run regressions across the paths they touched before shipping to the test environment, but the existing test cases carried extensive dependencies that made them difficult to run on demand. Second, new features suffered from a test case gap. To write tests for a new feature, that feature first had to be developed and deployed to the test environment, so that QA Automation Engineers had the selectors, labels, and other elements they needed in the Document Object Model (DOM). This forced QA Analysts and Engineers to context-switch to other work while waiting, and with so many features shipping in rapid succession, the result was mounting cognitive load and constant dependency wrangling. Third, the test scripts themselves were brittle. QA Automation Engineers typically received applications only after developers finished the code, always close to release. As they worked through Selenium and Playwright scripts, element IDs, classes, and other DOM and JavaScript attributes changed faster than they could fix them. Traditional automation was not augmenting the manual effort quickly enough to matter for a given release, and releases could slip as a result. Increasing the size of the QA team helped at the margins but did not address this root cause. First Orion realized they did not need more of the same. They needed a fundamentally different approach. Instead of writing code that describes how to navigate a UI, they wanted to describe what to test in plain English and have an intelligent agent handle the rest. That requirement led them to Amazon Nova Act. Why Amazon Nova Act fits First Orion’s requirements First Orion’s requirements came down to two problems: selector-based tests that broke as the UI changed, and the time it took to author new ones. Nova Act addressed both by letting QA Analysts describe tests in plain English rather than write and maintain code. When their AWS account team introduced them to Nova Act in March 2025, First Orion became a pre-release adopter and found value well before general availability. With Nova Act, First Orion describes what they want accomplished in natural language, not how to accomplish it. For example: “Log into the portal, navigate to billing, and verify the invoice total.” The agent reasons about the current UI state, identifies elements, and runs multi-step sequences autonomously. This was the critical difference from Selenium and Playwright. Those frameworks required QA Automation Engineers to write and maintain explicit element selectors that broke with every sprint. Nova Act instead reasons about what it sees on screen: labels, layout, and context. A Nova Act instruction like “click the Submit button” works regardless of the underlying CSS class. The model adapts to changing page layouts, handles dynamic content, dismisses pop-ups, and recovers from errors without human intervention. Developers can also interleave Python code, assertions, breakpoints, and parallelization directly with Nova Act commands. For First Orion, this meant three things: QA Analysts could author tests directly — no waiting for automation engineers to translate test cases into code. Tests survived UI changes — because Nova Act does not rely on fixed selectors, scripts no longer broke every sprint. No browser infrastructure to manage — AgentCore Browser handles provisioning, session recording, and parallel execution. How First Orion uses Nova Act First Orion’s main use case for Amazon Nova Act is testing their customer portal applications. They built an end-to-end system around the Amazon Nova Act SDK that takes QA analysts from English-language test case authoring through automated execution to integrated reporting. The following diagram shows the architecture, and we walk through each component below. Figure 1: Architecture overview of First Orion’s Nova Act test automation system The workflow begins with the Test Case Authoring UI, a React frontend where QA analysts browse, create, edit, and validate test cases in plain English. A custom templating engine handles dynamic variables (phone numbers, email addresses, business names) so the same test collection generates unique, realistic data on every run. Test collections are stored as JSON in Amazon Simple Storage Service (Amazon S3), which serves as the central repository. When a QA analyst triggers a test run, the Nova Act Test Runner picks up test cases from Amazon S3. This runner is a Python application on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate that orchestrates execution through the Amazon Nova Act SDK. The runner does not manage browsers directly; instead, it delegates browser operations to Amazon Bedrock AgentCore Browser. AgentCore Browser provisions managed browser instances, handles session recording, and supports parallel execution behind multi-factor authentication (MFA) without requiring changes to the target application’s auth flow. Browser profiles can start tests in a particular browser state, avoiding full login flows for every run. The Amazon Nova Act model receives natural language instructions from the test runner, reasons about the target web application’s current UI state, and executes the specified actions autonomously. Nova Act handles the browser manipulation on its own, the calling code does not need to manage selectors, waits, or page transitions. Results flow into an Allure reporting dashboard with a Microsoft Teams integration for notifications. First Orion relies on AgentCore for session recording and links video replays directly in their reports. The Microsoft Teams and Allure required custom integration. The Nova Act test runner used a Python script to orchestrate the SDK calls. Adopting this architecture has tightened integration between the QA and engineering teams. As more testing moves into the agentic flow, dependence on other vendors shrinks. First Orion also adopted the Kiro IDE for prototyping and development. Kiro’s built-in knowledge of AWS services and ability to read code bases made wiring everything up straightforward. The following figure shows an example test case as authored in the React UI. QA analysts write instructions in plain English (for example, “Navigate to the Billing page, verify the invoice total matches the expected amount, and download the PDF”). The Amazon Nova Act SDK translates these into browser actions at runtime without requiring element selectors or programming knowledge from the test author. Figure 2: Test case authored in plain English in the React UI The Allure dashboard provides test-level pass/fail status, execution timelines, and links to AgentCore Browser session recordings for debugging failed tests. Figure 3: Allure reporting dashboard Figure 4: Allure test results Before Nova Act, a QA Automation Engineer would manually identify DOM selectors, write scripts, debug brittle selectors, and iterate. This process can take days per test case. With this architecture, a QA Analyst takes the same English-language test case and runs it within minutes. The turnaround from test case definition to automated run dropped from days to minutes for supported scenarios. Results and impact While Amazon Nova Act is still in its adoption period and First Orion is adding features around it to fit their environment, they have seen as much as 20–25 percent reduction in QA cycles for specific types of testing. As coverage expands to additional application modules, they expect further efficiency gains. They estimate saving 25–30 percent of Engineering time with Nova Act, decreasing the amount of context switches between features. Engineering capacity that was previously needed to support the QA function has been redirected to feature engineering and value creation, allowing First Orion to focus on serving their customers better. Fully automated Nova Act runs allow for early regression detection and quick new feature testing, increasing their test coverage by upwards of 15 percent in some cases. By running tests against every build without waiting for manual QA availability, First Orion catches regressions earlier in the development cycle, before they reach production. Lessons learned and best practices Organizational buy-in matters. First Orion’s culture of trying n [truncated for AI cost control]