Pipeline watch
Most pipeline incidents start with a change. Pipeline watch notices the change and keeps a closer eye while it is fresh.
AvailabilityAvailable to everyone with early access to Varqeni.
Steps
- It starts on its own when the connector's configuration or the latest schema for the data topic changes, or when an application your administrator lists as writing to the source deploys a new release (a new image or spec, not a change in replicas).
- You can start one: press Start a 60 minute watch under the pipeline, or ask PelQi, for example watch the pipeline for 2 hours. You need to be on the on-call rota.
- It compares each check with the pipeline's own last seven days: heartbeat age, unconfirmed WAL, how many times the connector's tasks started in the last five minutes, and whether the pipeline is healthy. A downstream reader that stops or falls behind also counts.
- Drift is three checks in a row above twice the normal level (with a floor; for task restarts, two more than normal), or an unhealthy pipeline. It is told once in Teams, with the change that started the watch.
- When the watch ends it closes with a verdict, drift seen or no drift, kept with the others.
What you should see: Under the pipeline: Watching with the reason and end time, or No watch running with the last verdict and the baseline.
What it can and cannot do
Can
- Start a watch on its own after a connector config or schema change, or a new release of an application that writes to your source, or when you ask, and tell you once if the pipeline drifts, task restarts included.
Cannot
- Watch applications nobody listed as writing to the source, change anything, or judge drift before it has an hour of baseline.
If it does not work
- The baseline needs an hour of healthy checks before it is used. Until then only an unhealthy pipeline counts as drift.
Related articles
- Read the live pipelineThe Overview opens on your pipeline: the diagram first, then a panel with each part's key number, its recent history and the rate moving between parts.
- Ask PelQi a questionPelQi answers questions about what Varqeni sees, from its live evidence, and says when something is not in the evidence.
- Notices in Microsoft TeamsVarqeni posts a card to a Teams channel when it opens an incident, when the autopilot acts, and when a downstream reader needs a person.
Still stuck? Contact support or email info@fortisdatagroup.com.