OpenTelemetry logs and metrics
A service can answer every probe and still be failing: retrying behind the scenes, or with work piling up. Its logs and its own metrics show it first.
AvailabilityAvailable to everyone with early access to Varqeni.
How it works
- Logs: the Collector counts each service's log records at error severity or above. Varqeni reads only that count, per minute, never the text of a log. More than ten a minute (you can change it, for every service or one) names the service; after a change to its Deployment it names the change.
- Warnings are not errors: a burst of warnings names nothing.
- Metrics: for a metric a service sends itself, such as payments waiting, you set a rule: above or below a number. Crossing it names the service, with the value.
- One cause per outage: a service that is down or failing requests keeps that cause; its error logs are part of it, not a second fault.
- Setup: the Collector copies each service's name onto its log records and metrics, counts error records with its count connector, and exposes both to the Prometheus Varqeni already reads. Fortis sets it up with you.
- Fixes: none for these causes. Varqeni names the service and what it saw; a person decides.
- In the lab it named the right cause and service in 35 of 35 blind trials, with a warnings-only burst and the trace faults as controls.
What it can and cannot do
Can
- Name a service that logs errors above its threshold while it answers, and one whose own metric crosses a rule you set, from your OpenTelemetry Collector.
Cannot
- Read the text of a log, or run a fix for these causes.
Related articles
- OpenTelemetry tracesService Operations reads request errors per service, and which service calls which, from the traces your services send through an OpenTelemetry Collector.
- Read your services under ServiceUnder Service, the console shows each service you use, whether it answers and how fast, how many of its requests fail, which services it calls, your scheduled jobs, and what was deployed in the last 30 minutes.
- What each cause meansThe causes Varqeni names for a change data capture pipeline, in plain words.
Still stuck? Contact support or email info@fortisdatagroup.com.