Trust, but Verify: Testing an ERPNext Migration Through Its REST API

The validation problem

bench migrate completing is not the finish line, it’s the starting line of the real question: did the data survive? “Migrate exited 0” is necessary but nowhere near sufficient. What we actually needed to prove:

• Master data intact — customers, suppliers, items, chart of accounts
• Transactions intact — sales invoices, payment entries, journal entries (use frappe.client.get_count per DocType and compare against your v13 baseline)
• HR data intact — employees, salary structures, attendance records (the same count queries, cross-referenced against pre-migration exports)
• The app-split boundaries work — Payment Entries that reference Employees must still resolve now that Employee lives in HRMS and Payment Entry handling spans both ERPNext and the Payments app
• Writes work — reading old data is half the story; the new stack has to create documents too

Manual clicking catches some of this. Scripts catch all of it, repeatedly, on every re-run.

Why the REST API is the right tool

Frappe exposes a REST API for every DocType, and — the underrated part — for arbitrary server methods via /api/method/. Cookie auth is two calls:

curl -c cookies.txt -X POST http://localhost:8090/api/method/login \
  -H 'Content-Type: application/json' \
  -d '{"usr":"Administrator","pwd":"admin"}'

curl -b cookies.txt http://localhost:8090/api/method/frappe.auth.get_logged_user

From there, frappe.client.get_list, frappe.client.get, and frappe.client.insert cover ~90% of the validation surface — and because it goes through the same HTTP path a browser uses, every test also transitively verifies nginx, the backend, and site resolution. It is, functionally, an end-to-end test suite with a one-line setup.

The structure: ~43 checks grouped by category, each recording ID | category | test | status | detail. The console gets the table; a detail report gets every raw API response for post-mortems.

War story #1: the orphan false positive

The integrity check queries for Payment Entries whose party no longer resolves. First run: failures on every Employee-typed Payment Entry — each one pointing at an employee record that appeared to be missing.

Employees missing? No. The SQL only joined tabCustomer:

LEFT JOIN `tabCustomer` c ON c.name = pe.party
WHERE pe.party_type = 'Customer'  -- ← the bug: filter after the join

Payment Entry’s party field is polymorphic — party_type picks the target table (Customer, Supplier, or Employee). The check assumed single-target links and the plain “orphan” test was meaningless for it. This is worth dwelling on because it’s the shape of most migration validation bugs: the check itself is wrong, not the data.

The fix: one query per party type, each joining its own table:

SELECT COUNT(*) FROM `tabPayment Entry` pe
LEFT JOIN `tabEmployee` e ON e.name = pe.party
WHERE pe.party_type = 'Employee' AND pe.party != '' AND e.name IS NULL

Zero orphans across all three party types — including the cross-app cases (Employee now owned by HRMS). That’s the result that matters for a split-app migration.

War story #2: bash, backticks, and MariaDB

Same script, different layer of pain. The orphan queries live in a heredoc passed through bash -c inside docker compose exec. Python f-strings formatting SQL with backticks — FROM \tab{dt}`— hit a bash gotcha: **backticks inside double-quoted bash strings execute as command substitution**. The shell dutifully tried to runtab{dt}` as a command:

scripts/validate.sh: line 247: tab{dt}: command not found

The escapes needed to survive both bash and Python and produce valid SQL for MariaDB (which also refuses != '' parameterization quirks in some drivers) turned the query into unreadable soup. The maintainable fix: parameterized placeholders instead of string interpolation—

frappe.db.sql(
    """SELECT COUNT(*) FROM `{table}` si
       LEFT JOIN `{target}` t ON t.name = si.{field}
       WHERE si.{field} != %s AND t.name IS NULL""",
    ("",))

—and one bash quoting rule: heredocs that contain code use quoted delimiters (<<'EOF') so nothing expands anywhere. Rule two: don’t pass Python inside bash -c "..." at all if you can help it; pipe it in via stdin (docker compose exec -T backend bash -lc '...' <<EOF).

The final check script runs through bench-activated Python (source env/bin/activate, sites_path set), writes its records to disk, and leaves a zero-orphan receipt in the report.

The final suite

Forty-three checks. The categories map directly to what could break in a split-app migration:

The test script (scripts/test-api.sh) outputs a compact summary table to the terminal and writes every raw API response to backups/api-test-detail.txt. Run it before migration (against v13) to capture a baseline, then after migration to compare. The pre/post diff is the real validation.

War story #3: the CSS hash mismatch (the good one)

Everything green. Log into the desk, open a Sales Invoice and — the page renders as an unstyled wreck. Assets 404ing left and right.

Here’s the failure chain, reconstructed from container state:

1. The dev box’s bench build runs inside the backend container only. Builds produce content-hashed bundles (desk.bundle.VSDWJZOF.css) and rewrite assets.json, which maps bundle names → hashed URLs.
2. sites/assets/assets.json lives on the shared volume. The actual build artifacts land in the backend container’s overlay filesystem (the sites/assets/frappe path is a symlink into the container-local apps/ directory).
3. docker compose down && up to “fix” an unrelated issue → fresh containers from the image → the image’s original artifacts (different hashes: desk.bundle.CH3GYTYZ.css).
4. assets.json on the shared volume still points at the build’s hashes. Every desk page now requests CSS files that don’t exist. Every CSS request 404s. Unstyled desk.

The tell: backend and frontend listing sites/assets/frappe/dist/css/ and seeing different files — impossible on a shared volume until you follow the symlink into the container-local layer. Two containers, one volume, two filesystems.

The full fix, in order: recreate all containers (fresh image-state assets) → clear assets.json staleness → bench clear-cache → flush Redis (redis-cli FLUSHALL, since Frappe caches rendered asset lists in Redis) → hard-refresh. Root cause recorded the only way it stays fixed: never bench build in one container of a scaled stack — rebuild the image instead, which is what the image build (part 1) exists to do.

The lesson generalizes: in a stateless-Python-with-stateful-Redis design, asset maps are state, and state belongs in exactly one place.

The final tally

PASS: 42   FAIL: 0   SKIP: 1 (documented Payroll Period prerequisite)

The database is v15-native. HRMS owns the employees. Payments owns the gateways. Every cross-app link resolves. Custom fields survived the two-version gap. And the test suite that proved it is scripted, re-runnable, and already pointed at the next duty: running as a post-deploy smoke test for the ARM production install.