Introduction
Test exists to prove that the build satisfies the test suite that was frozen at the end of Design. This article explains how the suite is run in the sandbox environment, how test data is handled, and what happens when a test fails.
The suite that runs here is the approved one. It is not adjusted to suit the code that was produced, and that is what makes a successful result worth something.
The steps, in order
The stage moves through the following steps.
- The test data plan prepared during Design is applied, so the seed data the tests rely on is present in the sandbox.
- The frozen test suite runs against the deployed solution app in the sandbox.
- Results are reported test by test, so you can see what passed and what did not.
- If a test fails, ARVO can make a limited number of code-repair attempts and run the affected tests again.
- If the suite still cannot pass after those attempts, the run stops for human review.
- When the suite is green, the results and the supporting evidence are saved against the project.
[ Screenshot placeholder — the test results for a project, showing the test suite with passed and failed tests, demo tenant only, with no customer name visible. ]
The two types of test data
Seed data is the stable baseline. It is the master and setup information the tests need in order to have something to work with, and it remains available in the sandbox between tests rather than being rebuilt each time.
Transactional test data is different and it is created for an individual test, used by that test, and then rolled back when the test finishes. Nothing that the test created is left behind in the sandbox.
The effect of that split is practical. Because the baseline persists and the transactional data does not, every run starts from the same position and can be repeated with the same result, and the sandbox does not slowly fill with the residue of hundreds of test runs. Repeatable tests in a clean environment are what make a green suite meaningful, and they are also what makes it safe to run the suite as often as you need to.
The gate
The gate for Test is a green test suite. Every test in the approved suite has to pass, and when it does, the results and the evidence behind them are saved against the project.
The point that keeps the gate honest is that the suite itself does not change. ARVO can make a limited number of attempts to repair the code, but the approved tests stay exactly as they were frozen at the end of Design. A green result therefore means that the build satisfies what was agreed, not that the standard was lowered until the build could meet it.
Who approves or satisfies it
The gate is satisfied by the test run rather than by a person, and there is no sign-off at this stage; the human approval comes next, at Approve. If the run stops for human review, the decision is yours: either the build needs correcting, or the design was wrong and should be changed and approved again. What you do not do is quietly weaken the tests.
What is added to the project record
The test results and the evidence behind them are saved against the project, and the run appears in Runs with its outcome. Together with the approved designs and the training documents, this is the material an approver reads at the next stage, and it becomes part of the evidence pack the customer keeps at the end of the project.