Replacing Newman and Collection Runners with Spec-Driven CLI Tests

Quick answer: A collection runner like Newman is great at replaying a saved set of requests, but it tests the collection, not the contract. When teams want assertions that stay true as the API evolves, a spec-driven CLI that generates requests from the OpenAPI document and validates responses against it is the more durable choice. The migration does not have to be all at once: keep the collection runner for exploratory sequences while moving regression and contract checks to generated, schema-validated requests that fail the moment the API and the spec diverge.

Collection runners earn their place early. A developer records a sequence, adds a few assertions, and runs it headless in CI. The limitation shows up as the system grows: the collection is a frozen snapshot, so when the API changes, the collection keeps sending the old request and the assertion is often just a status code with no schema check. Duplicating every flow across environments and keeping the collection in sync becomes manual work. A spec-driven approach flips that, deriving the requests from a single source of truth so the tests and the contract cannot quietly drift.

Collection runner versus spec-driven CLI

The two tools optimize for different things, and naming the difference clarifies what to move.

Move a regression flow to generated requests

A spec-driven CLI can generate a request for an operation and validate the response against the schema, including status codes and body shape, without a hand-maintained collection.

# run an operation against an environment, validating against the spec
openapi-cli request \
  --spec ./openapi.yaml \
  --base-url https://api.staging.example.com \
  --operation createOrder \
  --body @./fixtures/order.json \
  --expect-status 201 \
  --validate-response

# the CLI verifies the response matches the documented schema,
# catching a changed field or missing property a status-only check misses

Keep assertions meaningful, not just 200

The biggest win of moving off a collection runner is schema-level validation. Assert both the status and the response contract, and for error paths validate the documented error shape. A generated check that the response conforms to the spec is far stronger than a collection assertion on a single hard-coded field.

it("returns a valid order that matches the contract", async () => {
  const res = await client.createOrder(makeOrderFixture());
  expect(res.status).toBe(201);
  // validates the full body against the Order schema, not one field
  expectValidAgainstSpec(res.data, "Order", "201");
});

it("rejects an invalid currency with the documented error", async () => {
  const res = await client.createOrder(
    makeOrderFixture({ currency: "xxx" })
  );
  expect(res.status).toBe(400);
  expectValidAgainstSpec(res.data, "Problem", "400");
});

Migrate incrementally, not by rewrite

Start with the highest-value regression flows and the error paths, generate those from the spec, and leave the collection runner for ad hoc exploration until the generated suite covers the critical paths. A local-first workflow keeps the OpenAPI document at the center and runs the generated checks in CI, so every build proves the implementation still matches the contract. The goal is not to remove a familiar tool for its own sake; it is to stop maintaining frozen request snapshots and let the spec drive the tests that protect the API.

Frequently asked questions

Do I have to abandon my existing collections?

No. Migrate regression and contract coverage first while keeping collections for exploratory work. The two can coexist until the generated suite covers what matters.

Can a spec-driven CLI handle multi-step workflows?

Yes, generated clients can chain operations and pass values between steps, with the added benefit that each response is validated against the schema along the way.

What about load tests and performance?

Keep dedicated load tools for performance; the spec-driven CLI is for functional and contract correctness. They serve different layers and can share the same fixtures.

How do I avoid generated tests that only assert status codes?

Enable response validation against the schema and add explicit assertions for business rules on top. Schema conformance plus a few targeted behavioral assertions is the pattern to aim for.

This is part of the Postman Alternatives Hub — start there for the full picture of local-first API workflows.