Data volume and schema drift
A pipeline can be up and still be wrong: rows written to the source that never reach the topic, a source that suddenly writes far less or far more, a column added by a release. Varqeni compares each window with what is normal for that pipeline, read-only.
AvailabilityAvailable to everyone with early access to Varqeni.
Steps
- What it reads: how many rows were inserted into the watched source tables (from PostgreSQL statistics), how many messages reached the topic, the tables' columns from the catalog, and the schema version registered for the topic. Nothing new is granted and nothing is changed.
- What it tells you: the source writing but much less reaching the topic, the source writing far less or far more than usual, and any column added, removed or retyped, each as a Data drift notice in Teams, and again when a volume problem clears.
- With an incident: when the pipeline breaks within an hour of a schema change, the investigation says so: "Schema changed before this", with how many minutes before.
- Volume is judged against a baseline Varqeni learns from healthy hours, so a new pipeline is not judged until it has one.
- In the lab it was right in 30 of 30 blind trials, with no notice on any trial where nothing changed.
What you should see: The pipeline view in the console shows any open drift finding and the latest window against its baseline.
What it can and cannot do
Can
- Tell you when the source writes but the topic does not keep up, when source volume collapses or spikes, and when the source schema changes.
Cannot
- Fix the publication, the load or the schema: it reports read-only, and a person decides.
Related articles
- Check an incident for lost or duplicated dataAfter a pipeline incident, ask Varqeni whether any rows went missing or arrived twice. It compares the source with the topic, read-only.
- The platform underneath a data incidentWhen a pipeline breaks because something it runs on broke, the investigation says so: the node, pod or service underneath, named beside the cause.
- Pipeline watchAfter a change, Varqeni watches the pipeline for an hour against its own baseline and tells you if anything drifts.
Still stuck? Contact support or email info@fortisdatagroup.com.