🔬Inside Testnova

The promise everything else serves

Anyone can generate test scripts with AI now. The hard part is trusting the answer. A tool that raises false alarms is worse than no tool at all: every false failure costs an engineer an afternoon proving the app was fine, and after the third one people stop reading the reports. A tool that passes tests it never really ran is worse still, because it hides the bugs you bought it to find.

So Testnova is built around one commitment: when it says FAILED, that is a real bug in your application. Not a broken selector. Not an expired login. Not the tool tripping over itself.

That is an architecture, not a slogan. Every outcome is sorted into one of five buckets, and only one of them, a genuine application defect, is ever allowed to show as red. A blocked login, a problem in the test script, an environment failure, or a gap in what Testnova could see about your app each gets its own honest label and its own next step. This document is the machinery behind that, service by service.

Testnova Auto

Auto is the flagship: point it at your app, describe a feature in plain English, and it produces acceptance criteria, test cases, runnable scripts, a real run against your app, and a report you can act on.

Underneath it is not one clever prompt. It is a stack of named mechanisms, each built to kill a specific way that AI-written testing goes wrong. They are grouped below by the stage they guard.

Auto: seeing your app, honestly

• Deep Snapshot. Before writing anything, Auto explores your app and records what is really there: real fields, real buttons, real wording. Passive crawling is not enough, because the interesting parts of an app hide behind buttons, so it opens forms and steps through wizards and tabs to reach them.
• Submit-and-Capture. It goes further and actually submits a bounded number of forms, because the confirmation message your app shows after a save is exactly what a test must check, and you cannot learn it by reading the page. This is the difference between a test that checks "no error appeared" and one that checks what your app really said.
• Explore Safety. It never breaks your app to learn about it. Delete buttons are opened only far enough to reveal the confirmation dialog, then cancelled. Commit, pay, checkout and logout are never clicked. Learning about your app must not change it.
• App Map. Mobile apps have no links to crawl, so Auto builds an inventory of your distinct screens and grounds every mobile test in that instead.
• Grounding Freshness. When your app changes, the old picture is marked stale and the affected tests re-verify against the new one, so a test is never quietly built on a screen that no longer exists.

Auto: agreeing what "correct" means

• Gap Questions. Some rules are invisible from outside: a maximum length, whether duplicates are allowed, what happens to a queued job when a limit is hit. A system that guesses those produces confident, wrong tests. Auto raises a question for you instead of inventing a rule. This is the never-fabricate discipline applied at the level of truth itself.
• Confirmed Oracles. Acceptance criteria are drafted in plain language, each with a concrete expected outcome, and you confirm them. Nothing is treated as truth until you agree it is.
• Harvest and Reconcile. Auto records the messages your app really emits during real runs, but an observed message never becomes the expected answer by itself. Facts are recorded; verdicts are issued separately against your confirmed criteria. Otherwise the tool would quietly learn your bugs as the standard.

Auto: designing the coverage

• Feature Analysis. Auto reasons about the feature before writing a case: the entities, the rules, the states, what should happen when things go wrong. One-shot generation over-tests one operation and misses whole others, such as edit, pagination, or cancelling a delete. Understanding first, writing second.
• Formal Design Techniques. Cases use what a senior QA engineer uses, not brainstorming: equivalence partitioning, boundary values, decision tables, state transitions. Most defects live at the edges, so that is where the cases go.
• Binary Oracle. Every case states a specific expected value or message. "No error appeared" is never treated as success, because the absence of an error is not proof of anything.
• Cap-as-Ceiling. The number you ask for is a ceiling, not a quota. If a feature genuinely warrants eight sharp cases, you get eight, not twenty padded ones.
• Use the Artifact. It checks that a record can actually be used after it is created, not merely that saving appeared to work. A record that saves but cannot be used is the defect your users hit most.

Auto: writing the scripts

• The Hard Rule. If Auto cannot ground a test in elements it genuinely captured from your app, it does not write the test. It tells you why, and what to do about it. A refusal is the product working: a script built on invented selectors looks like progress, then fails on its first run and sends an engineer hunting a bug that never existed. No script beats a fictional one.
• The Faithfulness Contract. Each test asserts exactly what its case promised, and nothing else. An unrelated extra assertion that fails turns a healthy app into a false alarm, which is the fastest way to lose your trust.
• Never Invent Data. IDs, names and records that must already exist in your database are taken from your app, never made up.
• Assert the Prevention. When your app is supposed to block something, a disabled button is the pass. Clicking a correctly-disabled control and waiting for a result would report a false failure against an app doing exactly the right thing.
• Blocked is not Failed. A precondition Auto cannot meet, an OTP it cannot receive, an environment it cannot reach, is reported BLOCKED with the reason. It is never dressed up as a defect in your app.
• Compensation. Every test cleans up after itself: records it creates are removed, changes are reverted, a password it changes is changed back. Cleanup never changes the verdict, and anything left behind is named explicitly.

Auto: verifying and self-healing

• Verify-Until-Green. A generated script is not "ready" because it looks right. Auto runs it against your real app, and only a script that actually ran green is presented as ready.
• Golden Canary. If the first test in a batch hits a login or session wall, the whole batch stops. You get one honest cause instead of a page of phantom failures, and no time is spent walking every remaining test into the same wall.
• Rule-Heal First. Obvious repairs, a renamed field, a moved page, a timeout set too tight, are made deterministically and for free before any AI is involved. Cheap and certain beats clever and expensive.
• The Anti-Fake-Heal Rule. Repair fixes tests, never verdicts. If an element genuinely disappeared, or the failure is not a selector problem at all, that surfaces as a real defect rather than being quietly patched green. Healing a real bug into a passing test would be the worst thing this product could do.
• No Guessing on Ambiguity. When a change could match more than one element, Auto does not choose. It shows you.
• Assertion Tightening, with an honesty guard. After a test passes, Auto can sharpen it to assert your app’s real message, but only when that message matches your confirmed expectation. Asserting whatever the app happened to say would bake today’s bugs in as tomorrow’s standard.
• Stale-Green Closure. Any edit to a script clears its previous pass. A changed test must re-earn its green, so unverified code can never reach your repository wearing an old tick.

Auto: running and reporting

• Suite Ordering. Read-only checks run first, then creates, then updates, with destructive and password-changing tests pinned last. An early failure cannot poison the tests behind it, and a credential change cannot lock the rest of the run out of your app.
• The Trust Doctrine, end to end. Every outcome lands in one of five buckets, and only a genuine application defect is ever shown as red. One place in the code makes that decision, so the dashboard, the report, the PDF and the email cannot disagree about what "failed" means.
• Judgment Rules. A large curated rule base decides defect versus pass versus unverified, and every rule exists to kill a class of false alarm. Two examples: the tester’s own limitation is never reported as your bug, and a "saved successfully" toast is never accepted as proof that anything persisted.
• One Root Cause. Failures sharing a cause are grouped into one, but only when the evidence supports it. When it does not, they stay separate rather than being tidied into a neat story.
• Two Gates to your Repo. A script reaches your repository only with your approval and a genuine passing run. Both, never one.
• Traceability. Requirement to case to run to bug, mapped end to end, so nothing is orphaned and no requirement is silently untested.

What Auto will not do

The refusals are the product, so they are worth stating plainly. Auto will not write a test for a screen it could not see. It will not report a blocked login as a bug in your app. It will not mark a test ready that has not genuinely passed. It will not heal a real failure into a green tick. It will not push a script to your repository without both your approval and a real passing run.

Each one is a place where a less careful tool shows you something green and moves on.

The other services

This document grows one service at a time. Explorer, Lens, API testing, Self-Heal, and Security each have their own safeguards, and each will get its own section here rather than a summary paragraph, because the detail is the point.

If there is a service you want documented next, tell us and we will write that one first.