Start with the decision the evidence must support
The first task in an evaluation is to define what the organization needs to decide. Is the machine suitable for a particular property? Is a manufacturer ready to support a regional channel? Can a process achieve the required output without consuming more supervision than expected? These are different questions and may require different evidence.
A useful test brief identifies the intended task, operating area, relevant conditions and acceptance criteria before the equipment arrives. It also records exclusions. A result obtained within a narrow environment should remain a narrow result until additional evidence supports expansion. Changing the acceptance standard after seeing the demonstration weakens the value of the exercise.
NIST's guidance on collaborative-robot integration begins with identifying which workcell is best suited to the technology. NIST AMS 100-41. The corresponding principle for field evaluation is to choose a real, bounded use case. Testing a machine against an undefined ambition cannot produce a clear go, revise or stop decision.
Record what the current process actually requires
A robot's performance needs a meaningful comparison. Document how the task is currently completed, including preparation, active work, inspection, rework and interruptions. If those activities are excluded from the human baseline but included in the robotic trial—or the reverse—the comparison will mislead the buyer.
The baseline should describe the result as well as the time. A property may require a particular finish, a warehouse a confirmed delivery, or an inspection process a usable record. Count acceptable output against that requirement. A faster pass that leaves unresolved work is not equivalent to a completed task.
Keep the comparison proportionate. An early trial does not require a complicated financial model, but it does require transparent assumptions. Record which staff time could actually be redeployed, which tasks would remain manual and what work is newly introduced by the robot. This prevents an attractive machine cycle from becoming an unsupported claim about whole-operation savings.
A demonstration is the beginning.
Define
Document the site, task and acceptable result.
Measure
Record task completion, interventions and elapsed time.
Stress
Test permitted changes in terrain, lighting and workload.
Recover
Observe safe stops, restarts and fault handling.
Support
Check parts, manuals, diagnostics and warranty responsibilities.
Decide
Compare evidence with the original acceptance criteria.
No completed ARA field results are claimed. Testing must remain within manufacturer instructions and an approved safety plan.
Build a matrix of conditions, not a collection of highlights
List the conditions that can materially change the result. For outdoor work, these may include surface moisture, terrain, vegetation, satellite visibility and seasonal conditions. For an industrial process, they may include load variation, lighting, packaging, route congestion and shift practices. The matrix should remain inside the manufacturer's permitted use and an approved safety plan.
Begin with representative ordinary conditions, then introduce controlled variation. Do not combine every difficult condition immediately: if a trial fails, the team needs to understand why. Once individual effects are understood, selected combinations can reveal interactions that separate tests miss.
NIST's response-robot methods illustrate the value of repeatable apparatus, procedures and performance measures for comparing capabilities. NIST: Ground Robot Tests. An original commercial evaluation can adopt that measurement discipline without claiming to have conducted a formal standardized test. State exactly which procedure was used, who observed it and which conditions were recorded.
Count completion, assistance and elapsed responsibility
At minimum, record scheduled tasks, attempted tasks, accepted completions and interventions. Define an intervention before testing: a remote instruction, physical repositioning, configuration change and technical repair are different events. Combining them into one number hides where the operational burden falls.
Time should be equally explicit. Separate autonomous work, charging, waiting for access, waiting for support and hands-on recovery. Track staff attention even when it occurs away from the machine. A short phone call repeated throughout a shift can disrupt other work without appearing in the robot's operating log.
Keep the denominator visible. A completion percentage means little without the number of attempts and their conditions. Small samples can reveal specific failures but cannot establish rare-event reliability. Report a limited trial as limited. Where outcomes differ sharply across conditions, preserve the separate results instead of averaging a demanding route together with an easy one until the problem disappears.
Make recovery part of the task
The most revealing event in an evaluation may be the first interruption. Can an ordinary trained operator understand the message? Does the documentation identify the correct response? Is the machine accessible? Does restoring operation require equipment, lifting assistance, a specialist or a network connection that is not normally available?
Recovery exercises should be planned with the manufacturer and competent safety personnel. They should not involve deliberately defeating safeguards or placing people in danger to create dramatic evidence. The objective is to verify permitted procedures and observe what happens when a representative interruption is handled correctly.
Measure the complete return to useful service. Restarting a motor is not enough if the map is lost, the work needs to be repeated or the operator must wait for support approval. Also inspect the surrounding result: did the interruption leave a load unsecured, an area unfinished or records incomplete? A robust deployment proposition includes the consequences of stopping as well as the capability to continue.
Control the configuration behind every result
Record the hardware version, attachments, relevant settings, software release and installation configuration. Photographs and logs should identify the tested setup without exposing unnecessary personal or confidential information. If the manufacturer changes the machine during the evaluation, label the resulting evidence as a new configuration rather than quietly combining it with earlier results.
NIST's connected-device software-update guidance includes authorized updating and verification of update sources. NIST: Software Update Capabilities. The field-evaluation implication is broader than installation security: an update needs a clear record and a reasoned decision about which prior tests must be repeated.
Improvement can be real while remaining unproven at scale. A revised release that resolves one fault should be checked for effects on adjacent tasks. Preserve both the original failure and the subsequent result. An evidence trail that records setbacks is more useful to a deployment team than a polished report in which every version appears to have worked equally well.
Evaluate the support system while evaluating the machine
A commercial trial should include installation instructions, maintenance access, replacement parts, warranty handling and technical escalation. Ask for the documentation an ordinary customer would receive. If the trial depends on direct access to an engineering team that will not support normal installations, identify that dependency explicitly.
Test the support process with a realistic, appropriately scoped issue. Can the provider collect useful diagnostics without repeated guesswork? Is ownership clear between manufacturer, distributor and installer? Are expected response times written down? The point is to learn how the relationship would function, not to manufacture a crisis or judge support from one unusually busy day.
Availability of a part should mean more than its appearance in a catalog. Establish the part identifier, order route, expected lead time and who can fit it. A potential channel partner must understand these obligations before promising a service level. Technical success and commercial readiness are related, but one does not automatically establish the other.
Earn the right to scale.
ARA’s intended approach. These are proposed gates, not completed milestones.
- 01
Evaluate
Define the task, environment and evidence needed.
- 02
Validate
Test performance and establish responsibilities with prospective partners.
- 03
Commercialize
Confirm customer fit, support capacity and full lifecycle economics.
- 04
Scale
Expand only as repeatable delivery and service capability are demonstrated.
Convert the evidence into a bounded decision
The outcome need not be a simple purchase or rejection. An evaluation may support a limited operating area, a revised installation, additional training, further seasonal testing or a decision to pause until a documented defect is resolved. Each conclusion should identify its evidence and remaining uncertainty.
Separate observed facts, supplier statements, evaluator interpretation and forecasts. For example, an observed successful task belongs in the first category; an estimated future service improvement belongs in the last. Neither should be rewritten as an established capability of the product currently available.
NIST's AI Risk Management Framework emphasizes continuing management as systems and contexts evolve. NIST AI RMF Core. The corresponding operational lesson is that acceptance is a controlled starting point, not the end of measurement. Agree on what would trigger reassessment: a major update, a different environment, changed payload, deteriorating recovery time or an expansion of responsibilities.
ARA's intended philosophy: evaluate, validate, commercialize, scale
For ARA, hands-on evaluation is intended to precede commercial expansion and distribution introductions. The proposed sequence is to evaluate whether a relevant capability exists, validate it against a defined use case, establish viable support and economics, and only then consider repeatable scale. These stages describe an intended method, not completed ARA tests or existing service capacity.
The near-term focus on residential and professional property-maintenance equipment makes seasonal and site-specific evidence especially important. A successful dry-weather mowing trial cannot settle winter performance. An owner-operated trial does not establish that a regional service network can support many customers. Each decision needs evidence at the scale and under the conditions it concerns.
This feature explains how to produce useful evidence. Its companion, Why Robotics Deployments Succeed or Fail, examines how organizations use—or fail to use—that evidence after installation. The difference is consequential: a disciplined test can reveal the right limits, but sustained operation still requires people, commercial arrangements and management decisions that respect them.
Sources & reading
Primary references used for this feature. Linked research and manufacturer accounts retain their original scope; they are not ARA performance claims.
- NIST AMS 100-41 — Cobot Integration Best Practices
Primary 2021 workcell-selection guidance; used for a narrow application-selection principle.
- NIST — Ground Robot Tests
Repeatable capability measurement; the article does not claim ARA has performed ASTM or NIST testing.
- NIST IoT Catalog — Software Update
Primary guidance on authorized and verifiable update mechanisms.
- NIST AI RMF Core
Lifecycle risk-management perspective, distinguished from a specific machine acceptance test.
This feature is original ARA editorial analysis. Examples describe the wider robotics industry; they do not imply an ARA supplier appointment, tested product, customer deployment or government relationship. Forecasts and proposed evaluation methods are distinguished from reported results.
Sources, methodology & corrections ↗