Вызван получатель с уже уведомленными разрешенными оповещениями

Question

Вызван получатель с уже уведомленными разрешенными оповещениями

У нас проблема с разрешенными оповещениями, которые отправляются слишком много раз. Мы используем один экземпляр Прометея и один экземпляр менеджера оповещений (без кластера).

Мы ожидаем получить сообщение от диспетчера оповещений с отключенными серверами (оповещения в состоянии активации) и вновь доступными серверами (оповещения в разрешенном состоянии). Что особенно интересно, это когда все серверы снова доступны (группа оповещений в разрешенном состоянии).

Когда сервер или несколько серверов снова становятся доступными, мы получаем несколько вызовов с разными разрешенными оповещениями. Мы должны получить уведомление о разрешенной информации только один раз. У нас есть только одно предупреждение, полученное с помощью сообщения (даже если два оповещения разрешены, см. Последнее действие).

Системная информация: Linux 3.10.0-957.1.3.el7.x86_64 x86_64
Версия Alertmanager: 0.16.1
Версия Прометея: 2.7.1
Файл конфигурации Alertmanager:

global:
  resolve_timeout: 8737h

route:
  group_by: ['alertname']
  group_wait: 0s
  group_interval: 60s
  repeat_interval: 8737h
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://localhost:9991/v1/alerts/'
    send_resolved: true

Пожалуйста, обратите внимание, что:

мы используем пользовательский экспортер, который запускает сценарий для проверки состояния каждого сервера
нам не нужно разрешать тайм-аут повторного интервала, поэтому мы используем большое значение, как предлагается здесь: #935)
- Файл конфигурации Prometheus:

global:
  scrape_interval: 5s
  evaluation_interval: 20s
scrape_configs:
  - job_name: 5c5d8e41c680703f3a26fb68-monitoring-1
    scrape_interval: 60s
    scrape_timeout: 10s
    metrics_path: /metrics
    static_configs:
      - targets:
          - server01
          - server02
          - server03
          - server04
          - server05
    relabel_configs:
      - source_labels:
          - __address__
        target_label: __param_target
      - source_labels:
          - __param_target
        target_label: instance
      - target_label: __address__
        replacement: 'http-scenario-exporter:9900'
rule_files:        
  - rules/5c5d8e41c680703f3a26fb68-monitoring-1.yml

Файл правил

groups:
  - name: 5c5d8e41c680703f3a26fb68-monitoring-1
    rules:
      - alert: 5c5d8e41c680703f3a26fb68-monitoring-1-0
        expr: >-
          http_scenario_ok{job='5c5d8e41c680703f3a26fb68-monitoring-1'}
          == 0
        for: 60s
        labels: {}
        annotations: {}

Вот как мы воспроизводим эту проблему (обратите внимание, что тег instanceName установлен экспортером).

server01, server02, server03, server04, server05 работают.

Я останавливаю server05:

{
        "level": 10,
        "time": "2019-02-11T13:51:25.895Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server05",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:51:25.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "instanceName": "server05",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
        },
        "_commonAnnotations": {
        },
        "v": 1
}

Я останавливаю server04:

{
        "level": 10,
        "time": "2019-02-11T13:57:45.994Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server04",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:57:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server05",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:51:25.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
        },
        "_commonAnnotations": {
        },
        "v": 1
}

Я перезагружаю сервер5:

{
        "level": 10,
        "time": "2019-02-11T14:01:25.998Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server04",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:57:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "resolved",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server05",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:51:25.888Z",
                "_endsAt": "2019-02-11T14:01:25.888Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
        },
        "_commonAnnotations": {
        },
        "v": 1
}

Я останавливаю server03: (здесь я получаю два вызова (!): Один с срабатыванием server04+server03, а другой с срабатыванием server04+server03 и сервер05 разрешен!)

{
        "level": 10,
        "time": "2019-02-11T14:03:46.001Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server04",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:57:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server03",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T14:03:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
        },
        "_commonAnnotations": {
        },
        "v": 1
}



{
        "level": 10,
        "time": "2019-02-11T14:04:06.002Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server04",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:57:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server03",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T14:03:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "resolved",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server05",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:51:25.888Z",
                "_endsAt": "2019-02-11T14:01:25.888Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
        },
        "_commonAnnotations": {
        },
        "v": 1
}

Я перезапускаю server04: (здесь я получаю много звонков (!). Всегда при срабатывании server03. Иногда с server04 разрешается, а иногда с server05 разрешается.)

{
        "level": 10,
        "time": "2019-02-11T14:07:46.002Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "resolved",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server04",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:57:45.888Z",
                "_endsAt": "2019-02-11T14:07:45.888Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server03",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T14:03:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
        },
        "_commonAnnotations": {
        },
        "v": 1
}


{
        "level": 10,
        "time": "2019-02-11T14:08:06.004Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server03",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T14:03:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "resolved",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server05",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:51:25.888Z",
                "_endsAt": "2019-02-11T14:01:25.888Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
        },
        "_commonAnnotations": {
        },
        "v": 1
}



{
        "level": 10,
        "time": "2019-02-11T14:09:06.005Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "resolved",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server04",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:57:45.888Z",
                "_endsAt": "2019-02-11T14:07:45.888Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server03",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T14:03:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
        },
        "_commonAnnotations": {
        },
        "v": 1
}







{
        "level": 10,
        "time": "2019-02-11T14:09:26.004Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server03",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T14:03:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "resolved",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server05",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:51:25.888Z",
                "_endsAt": "2019-02-11T14:01:25.888Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1",
        },
        "_commonAnnotations": {
        },
        "v": 1
}



{
        "level": 10,
        "time": "2019-02-11T14:10:26.005Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "resolved",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server04",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:57:45.888Z",
                "_endsAt": "2019-02-11T14:07:45.888Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server03",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T14:03:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
        },
        "_commonAnnotations": {
        },
        "v": 1
}





{
        "level": 10,
        "time": "2019-02-11T14:11:46.005Z",
        "pid": 1,
        "hostname": "8f80b4c90c57",
        "module": "DispatcherService",
        "method": "dispatchAlert",
        "_receiver": "web\\.hook",
        "_status": "firing",
        "_externalURL": "http://5743407ed2a5:9093",
        "_version": "4",
        "_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
        "_alerts": [{
                "_status": "resolved",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server04",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T13:57:45.888Z",
                "_endsAt": "2019-02-11T14:07:45.888Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        },
        {
                "_status": "firing",
                "_labels": {
                        "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                        "datacenter": "central",
                        "instanceName": "server03",
                        "job": "5c5d8e41c680703f3a26fb68-monitoring-1"
                },
                "_annotations": {
                },
                "_startsAt": "2019-02-11T14:03:45.888Z",
                "_endsAt": "0001-01-01T00:00:00.000Z",
                "_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
        }],
        "_groupLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
        },
        "_commonLabels": {
                "alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
                "datacenter": "central",
                "job": "5c5d8e41c680703f3a26fb68-monitoring-1",
        },
        "_commonAnnotations": {
        },
        "v": 1
}

...

0

prometheus prometheus-alertmanager

Источник

user856256 14 фев '19 в 09:09

0 ответов

Другие вопросы по тегам prometheus prometheus-alertmanager