Alerts integration on your validator (full node)

Hi friends, today I want to share with you my guide on how to integrate the alerts from Grafana into many platforms. That way, if something goes wrong with your validator, you can always be the first to know about it. :upside_down_face:

You need Grafana installed on the server (it’s a good service for monitoring the health of a node). If there are some questions about it, I will prepare guideline later.

We will use Pager duty and e-mail to accept the Alert. You should know that you can also use telegram bots or other programs to receieve messages. How to sign up in pager duty. 2

The guide consists of several parts.
Part 1. Creating integration key in pager duty
Part 2. Preparing the Alert itself in Grafana.

1.1 In your Pager duty app go to - services

Рисунок3

Рисунок3844×467 50.4 KB

1.2. Push “New service”

1.3 Name it

Рисунок15

Рисунок15873×485 50.9 KB

1.4 Push “Generate new escalation policy”

Рисунок4

Рисунок4868×483 51.8 KB

1.5 Select Events API v2

Рисунок6

Рисунок6864×483 111 KB

1.6 Copy your Integration Key

We created the integration key that we will need to connect our monitoring system with our messenger.

2.1 Go to your Grafana. Alerting - Contact Points.

Рисунок8

469×1140 97.4 KB

2.2 Push New contact point

Рисунок7

Рисунок71373×584 103 KB

2.2.1 Write any name

2.2.2 Select the service you want to be notified to from the list (Pager Duty or E-mail as you wish)

2.2.3 Insert the Integration Key you created in section 1.

2.2.4 You can test it by clicking on the “Test” button. A test message will come.

2.2.5 Save the contact point.

2.3 Remember you can just add your e-mail and you won’t need pager duty or telegram.

Рисунок9

Рисунок9917×523 55.7 KB

2.4 Next, we need to add our Contact point to the default Notification policies

Рисунок10

Рисунок101001×471 60.2 KB

2.5 Now we have to go to our dashboard

2.6 Choose the metric which defines the validator work (e.g. CPU load in % or fails in your proposals)

At the same time you can add alerts for other parameters yourself, if necessary. I will show you some metrics I’ve added for my validator.

It’s simple to find metric in your dashboard (1) and then push arrow (2) and edit

Рисунок11

Рисунок11844×488 67.4 KB

2.7 Then switch to alert at the bottom and create a new rule

Рисунок12

Рисунок12930×398 45.3 KB

2.8 Add Expression (button +Add Expression)

Set the rule: When CPU load is above 70%, then an alert will be triggered and I will receive an email notification (or Pager Duty, what you decide) Next, save.

Рисунок13

Рисунок131011×497 113 KB

2.9 Then we select the folder where the rule will be stored (GrafanaCloud)
The name of the group can be any. You can also specify the text that will be in the message. Edit field Description. Be sure to save.

Рисунок14

Рисунок14756×633 77.7 KB

I will also give you some screens how you can add alert on:

2.10 Failed proposal (will alert when there is at least one failed proposal)

Скриншот 11-02-2023 111112

Скриншот 11-02-2023 1111122203×1203 147 KB

2.11 Proposal count

Скриншот 11-02-2023 111337

Скриншот 11-02-2023 1113372253×1079 132 KB

Hope you found something useful for you! Feel free to ask any question :slight_smile:

12 Likes

Your efforts are truly appreciated,thank you

4 Likes

very good summary, thank you

3 Likes

Great insight :+1:

Good stuff. Thanks fren

1 Like

Great stuff fam :+1:

this is a good update my friend