Check an incident for lost or duplicated data
A data check answers the question people ask after a change data capture incident: did every row the application created reach Kafka, exactly once? Varqeni compares the primary keys in the source database with what reached the topic, over the incident from ten minutes before it opened to two minutes after it recovered. In the lab it was right in 24 of 24 trials against real faults: a paused connector (nothing lost), a table dropped from replication for a minute (the lost orders named by id, to within a row or two at the edges of the outage) and a forced second snapshot (duplicates).
Steps
- By itself: once a pipeline incident recovers and its window has settled, usually within five minutes, Varqeni runs the check on its own, signed Varqeni (automatic). It tells the team only when rows were lost or duplicated, and only once for the same rows; a clean answer is recorded and shown in the incident.
- To check by hand, open the incident and find Data check under the response buttons.
- If you are on the on-call rota, choose Check for lost data. The watch service runs the check on its next pass, usually within a minute.
- Read the answer: Nothing lost with the number of rows reconciled, Rows lost with their ids, or Duplicates with how many rows arrived more than once and whether a second snapshot sent them.
- Each check is sealed with the evidence, like an investigation, and appears in the list of investigations as a data check.
What you should see: Varqeni reads only the primary key of each row, through its read-only database role, and reads the topic through a ledger that consumes it without a consumer group. It never re-sends rows: when rows are missing it says what people usually do (an incremental snapshot of those ids, or a fresh snapshot), and a person decides.
What it can and cannot do
Can
- Compare the order ids your source database holds for a window with the ones that reached Kafka, and name any that never arrived or arrived twice.
- Say when it cannot tell, for example while the incident is still open or the newest events are still arriving.
- Check by itself after every pipeline incident recovers, and tell the team only when rows were lost or duplicated.
Cannot
- Re-send lost rows or remove duplicates. It says what people usually do, and a person decides.
- Read anything of a row except its key.
If it does not work
- "Not checked: the incident has not recovered yet": wait until the pipeline is running again, then check.
- "The window has not settled": the newest events are still arriving. Check again in a minute.
- "Kafka has deleted older ones": the topic no longer holds events from that time, so the window cannot be checked against it.
Related articles
- The incident timelineEverything that happened in one incident, in order, with who did it: from the first signal to the system being healthy again.
- Post-incident write-ups and follow-upsA blameless write-up drafted from the record, ready for you to check, and a follow-up ticket in Jira in one step.
- What each cause meansThe causes Varqeni names for a change data capture pipeline, in plain words.
Still stuck? Contact support or email info@fortisdatagroup.com.