Troubleshooting Missed blocks

Opening this topic in the forum as a place to discuss this issue and hopefully get some ideas.

Looking in Hubble, many validators occasionally miss a block. How should an operator diagnose missed blocks? There is a wealth of data in the blockchain, and in logs that we keep on our validators and our sentry nodes. What should we be looking for?

Does the Tendermint/Cosmos team have expectations for what this should look like in the production network? If there are 100 validators distributed globally & 5 second block times, is it expected that all properly operating validators will have 100% uptime? Or is it expected that a properly operating validator will miss some percentage of blocks? Is there any theory that the team can share that would help the validator community understand the dynamics?

Also very interested in this. I missed a block on gaia-7003

Yeah, great question. I think it could also be helpful if people posted how they have their validator/sentrys setup. To know if it’s related to some kind of specific architecture.

This is not a solution you are asking for but, you can look up /commit?height= on any well-connected full-node for any validator’s block signing status. When I test this, I found out we should check /commit?height= at least 2 blocks before the current height because it sometimes updated late. Below is an implementaion.

dlguddus/cosmos_api_tools/blob/master/validator_monitor.py

# -*- coding: utf8 -*-

# monitor validator's commit activity & alert by telegram message
# by dlguddus(B-Harvest)

import time
import requests
import json
import os
import threading
import sys
import datetime
from flask import Flask
from flask import Markup
from flask import Flask
from flask import render_template

validator_address = "" # put your validator hex address here
telegram_token = "" # put your telegram bot token here
telegram_chat_id = "" # put your telegram chat_id here

How should an operator diagnose missed blocks? There is a wealth of data in the blockchain, and in logs that we keep on our validators and our sentry nodes. What should we be looking for? This is a great question but it can be difficult to answer without having access to logs across many validators. Here are some general notes: • A proposer will wait timeout_commit after seeing a commit for a block before proposing the new block, to give time for more than the required +2/3 votes to get in. This currently defaults to 5s, but it used to be 1s so its possible some validators still have the old setting. • We distinguish between canonical and non-canonical commits. A non-canonical commit is the first +2/3 you’ve seen for a block. A canonical commit is the actual set of votes for block H that get included in H+1. A canonical and non-canonical commit for the same block intersect in at least +1/3 of voting power. See the `canonical` field in the `/commit` response. We could do a better job of exposing the non-canonical commit seen first by each node to get a sense of how the votes were propagated (currently, you’d have to just keep pinging for /commit on each node for the…

Excerpt (1197 of 3575 characters). Read the whole post on the forum ↗