Skip to main content

Error tracking

Error tracking is one row per distinct failure seen across automation runs, with a count of how often it has occurred — rather than one row per run, which is what the run list already gives you.

:::info Plan Error tracking is a Business plan feature. The link is hidden on lower plans, the route refuses, and the endpoint refuses again for itself. :::

How failures are grouped​

Two failures are the same error when their first line matches — the stack's first line where there is a stack, and the message otherwise.

This is deliberately crude, and it is crude in the right direction. The rest of a trace varies per run: line numbers shift, timestamps and ids appear, paths differ between machines. Matching on the whole thing gives one row per run, which is the list you already had.

What a row tells you​

  • The failure's first line, which is the identity
  • How many times it has been seen
  • When it was last seen, and in which run
  • Which test it came from

Where the rows come from​

Only ingest writes them. They are created as reports are parsed, guarded on results having actually been stored — so a re-sent report cannot inflate the counts. The screen itself reads and does nothing else; there is no way to edit or hand-create an error.

Old runs uploaded before error tracking existed can be backfilled by an administrator with scripts/backfill-automation-tests.js, which does the same work for reports already in the system.

Using it​

The value is in the count. A failure seen twice is noise; a failure seen sixty times across three weeks is a bug in the product or a genuinely broken test, and the count is what tells the two apart from a list of runs that all look equally red.

Sort by occurrences, take the top row, and fix that.