Monitoring/Alerting for your Validator

This post is a discussion about what metrics are important to monitor/alert on when running a validator. The following are places where metrics can be obtained and some notes on those locations:

• Prometheus Port (26660)
  • Number of network peers (p2p_peers) can be alerted on if below a threshold
  • Time between blocks (consensus_block_interval_seconds) can be alerted on if above a threshold
  • Many of the other statistics are useful for tracking validator profitability and should be tracked such as:
    • consensus_validators_power
    • consensus_num_txs
• RPC Port (26657)
  • The /health endpoint can be used as a confirmation that the process is still running

Would be interested in some feedback from @ajc and the rest of the Figment team on this post as they are working on alerting and monitoring in their hubble tool.

We think there will be a couple different kinds of alerts and long running metrics to track.

A web app comparison: you definitely want to monitor your servers closely for uptime and error logs, and you also use a tool like New Relic to track more complex and longer ranging metrics.

We want Hubble to be more like New Relic - surfacing more complex things in near real-time. Stuff that would be difficult for every Validator to reimplement and run on their own infrastructure.

Hubble currently syncs the blockchain once per minute and processes new blocks in less than a second, so it won’t be instantaneous, but should be fast enough.

The initial version of Alerts was built around this spec:

Event types

1. In and out of Validator set
2. M of N pre-commit votes missed
3. N in a row pre-commit votes missed
4. Voting Power changed by N percent

Notification types

1. Instantaneous
2. Daily summary

Instantaneous notifications will only be sent once every 15 minutes per-user per-validator.

So that’s what we’ve got so far. Curious to hear what kind of alerts and metrics are interesting for others!

ajc:

A web app comparison: you definitely want to monitor your servers closely for uptime and error logs, and you also use a tool like New Relic to track more complex and longer ranging metrics.

Just expanding on that thought a little bit. Hubble will alert for some things that we think are highly relevant to a us as validator, and hopefully to others, but it doesn’t help diagnose or provide any general health monitoring. We don’t think Hubble is a replacement for internal monitoring tools. Normal system monitoring, network health, security, ID and monitoring specific to gaiad on each node will be necessary. We think an external monitoring system like Hubble is an important complement to internal tools, because it tells us how our validator is performing from the perspective of the blockchain.

Hello, the “Engineer”!

@suyu good job!
will you share with us

Good point, but I have one question for the others validators, wich program to monitor are you using to? I’ve read about Graylog, Zabbix and ELK; have any recommendations?
thanks!!

@suyu Can you share that dashboard code? Would love for the community to have an easy Grafana dashboard for the prometheus metrics!

ELK is a great choice for the logging portion of it. As far as metrics I would recommend either Prometheus (what that grafana dashboard up there is pulling it’s data from), or InfluxDB which provides a similar feature set.

I’ll read about them, thanks!

Wow ! That’s so cool. Good job @suyu

I created a very simple dashboard for Grafana using the Prometheus metrics. Uploaded it to Grafana, so anyone can use it. Here is the link: https://grafana.com/dashboards/7044
Feedback on useful metrics welcome!

This is awesome @katernoir! I’ll give it a try!

simple??? I think it’s not simple
great work!!

This is awesome! Great work @katernoir!

I’m trying to set up Grafana/Prometheus and I feel like I’m missing something. I set prometheus = true in config.toml. I checked that it’s listening on port 26660, and it appears Grafana is connecting: netstat -an |grep 26660 tcp 0 0 127.0.0.1:41142 127.0.0.1:26660 ESTABLISHED tcp 0 0 127.0.0.1:41146 127.0.0.1:26660 ESTABLISHED tcp6 0 0 :::26660 :::* LISTEN tcp6 0 0 127.0.0.1:26660 127.0.0.1:41142 ESTABLISHED tcp6 0 0 127.0.0.1:26660 127.0.0.1:41146 ESTABLISHED However, in the dashboard I see this: Screen Shot 2018-07-19 at 11.06.50 PM.png 2416×982 136 KB And I see lots of 503 errors in the logs when the dashboard is running: t=2018-07-20T03:09:37+0000 lvl=info msg=“Request Completed” logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/api/v1/query status=503 remote_addr=127.0.0.1 time_ms=1 size=59 referer=“ http://localhost:3000/d/ajjGYQdmz/cosmos-network-dashboard?refresh=5s&orgId=1 ” t=2018-07-20T03:09:37+0000 lvl=info msg=“Request Completed” logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/api/v1/query status=503 remote_addr=127.0.0.1 time_ms=4 size=59 referer=“…

Excerpt (1174 of 1254 characters). Read the whole post on the forum ↗

You need to run Prometheus to monitor the 26660 target by editing the prometheus.yml. It will listen at port 9090. You then point your datasource in Grafana to <your_address_running_prometheus>:9090

The config in prometheus.yml can be as simple as this.

image.jpg790×276 63.2 KB

We’ve turned on Hubble Alerts and Events for gaia-7001.

Instructions for how to use and subscribe are here:

Hubble Alerts and Events

You can now subscribe to receive Alerts when Events happen to a Validator. Event Types Currently there are four Event types: Voting Power Change % (configurable) Misses N of Last M Precommits Joined/Left the Active Set Misses N Consecutive...

@katernoir Do you plan on updating this for 7004?

@kwunyeung How about the telegram bot for 7004?

You mean the Grafana dashboard? It should work with any network your node is running, as long as the connection to Prometheus doesn’t get changed. So no need to update it I haven’t tried it in 7004 yet, though.

We have updated it and I’m testing with it now. I keep receiving absent validator notification if our validator node didn’t send vote to a certain height. You can add the bot and subscribe to your validator address to try.

GaiaBot

You can contact @ForboleGaiaBot right away.

Where is prometheus.yml?

Thanks for your work on this! I’m trying to configure the dashboard to monitor one of my validators. Could you please provide some guidance on the required settings for the Prometheus data source, show in the screenshot?

Screen Shot 2018-08-02 at 12.50.01 PM.jpg1538×1586 167 KB

The URL in the HTTP section needs to be configured to the HTTP API URL of the Prometheus server that is scraping your validator input.

The Grafana documentation on this topic might be helpful as well.

Can you share your prometheus.yml so that we can understand the setup better

hey, try switchin in data sources in grafana from server to browser.

Because I’ve been asked this a lot, I provided a smat step-by-step instruction to setup Grafana with my dashboard. Hope this works and will help people to get started. • Step: Install Grafana ( http://docs.grafana.org/installation/debian/ ) & start it • Step: In .gaiad/config.toml set prometheus=true • Step: Restart gaiad to apply config changes • Step: Download prometheus ( https://prometheus.io/docs/introduction/first_steps/ ), edit prometheus.yml Add the following: ``` # COSMOS MONITORING # The job name is added as a label `job=<job_name>` to any timeseries scraped$ - job_name: 'cosmops' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['localhost:26660'] labels: group: 'cosmops' ``` • Step: start prometheus with: ./prometheus --config.file=prometheus.yml • Step: Open Grafana in Browser & Do initial Setup • Step: Under Configuration -> Data Source -> Add a new Data Source Name: CosmosDataSource Type: Prometheus URL: http://localhost:9090 Scrape Interval: 5s Rest is Default -> Save&Test should add DataSource • Step: In Grafana goto Dashboard -> Import • Step:…

Excerpt (1194 of 1340 characters). Read the whole post on the forum ↗

I’m working on some monitoring and alerting for validators and sentries -

1 - Using Icinga for alerts

2 - Updating a Grafana/Prometheus dashboard

3 - Log analysis

I plan to open source the tools when they’re ready.

For starters, I’m wondering if anyone has done any research into log patterns that indicate missed pre-commits?

Adding feedback from -

@mattharrop If set to do so, gaiad will write every signature in each block to syslog. Just check for your validator’s ID in the block of signature, if it’s not there, that’s a miss.

@haasted We’ve created a tool to monitor for pre-votes. https://github.com/validator-network/votewatcher Feedback welcome

In our old gaiabot, it utilizes a systemd package to monitor the journal. If you run your gaiad as systemd service, then the journal can be received from it. You may take a look.

forbole/bot_monitor/blob/master/respond/botstoredb.py

#!/usr/bin/env python
from __future__ import division
from systemd import journal
import re
import sys
import pymongo
from pymongo import MongoClient
import json
import os
import subprocess
import requests
import time
from requests_futures.sessions import FuturesSession

db_info = sys.argv[1]
loginData = "DATABASE_CONNECTION"
db = MongoClient(loginData)
client = db.BotData
collection = client.ValAddrID

However, I don’t quite like this approach as it uses a lot of resources to keep checking every line of journal log of the process to decide if it should send out an alert message.

Feedback from @jack https://twitter.com/jack_zampolin/status/1115987603243683841

A Grafana dashboard compatible with all the cosmos-sdk and tendermint based blockchains: https://github.com/zhangyelong/cosmos-dashboard

image.jpg1860×946 216 KB

image.jpg1859×553 102 KB

This post of mine is almost 2 years old. I’m not keeping this up-to-date anymore. Please find some more recent information.