I am trying to find out nodes left or nodes joined in ElasticSearch 6.4

Looks like node_left and node_joined events has been removed from marvel indexes in latest versions .
Is there any other way to create a watch to alert when a node joins or leaves the cluster? Besides doing a query for total nodes?

Any suggestions please?

If you are indexing your elasticsearch logs in a dedicated monitoring cluster, then you could create a watcher script in monitoring cluster that queries for specific keywords like added, removed.

[2019-09-12T12:15:56,802][INFO ][o.e.c.s.ClusterService   ] [1.1.1.1] removed {{2.2.2.2}{_bm_BBMQQJCCVx6HbuBa_B}{n83UoTCLSiWOyfst-a_s0w}{2.2.2.2}{2.2.2.2:9300}{zone=sandeep-node, ml.enabled=true, tag=sandy-test},}, reason: zen-disco-receive(from master [master {3.3.3.3}{6V0mpFS4RXyE4K11eb1Iyg}{Q6uvX9ySQ5q00eZfme2cHA}{3.3.3.3}{3.3.3.3:9300}{ml.max_open_jobs=10, ml.enabled=true, tag=sandy-test, zone=sandeep-node} committed version [559]])
[2019-09-12T12:16:13,898][INFO ][o.e.c.s.ClusterSettings  ] [1.1.1.1] updating [cluster.routing.allocation.enable] from [none] to [all]
[2019-09-12T12:16:37,892][INFO ][o.e.c.s.ClusterService   ] [1.1.1.1] added {{2.2.2.2}{_bm_BBMQQJCCVx6HbuBa_B}{RToPW_2dQw2vqpMOYpLlTg}{2.2.2.2}{2.2.2.2:9300}{ml.max_open_jobs=10, ml.enabled=true, tag=sandy-test, zone=sandeep-node},}, reason: zen-disco-receive(from master [master {3.3.3.3}{6V0mpFS4RXyE4K11eb1Iyg}{Q6uvX9ySQ5q00eZfme2cHA}{3.3.3.3}{3.3.3.3:9300}{ml.max_open_jobs=10, ml.enabled=true, tag=sandy-test, zone=sandeep-node} committed version [580]])

Below is a sample watcher script that queries for ERROR in the index *-eslogs-*. Elasticsearch logs are indexed into the said index.

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "*-eslogs-*"
        ],
        "types": [],
        "body": {
          "size": 1,
          "query": {
            "bool": {
              "must": [
                {
                  "query_string": {
                    "analyze_wildcard": true,
                    "query": "Level: ERROR"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m",
                      "lte": "now",
                      "format": "epoch_millis"
                    }
                  }
                }
              ],
              "must_not": []
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 1
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "from": "Watcher Alert <[email protected]>",
        "to": [
          "[email protected]>, [email protected]>"
        ],
        "subject": "Watcher Notification",
        "body": {
          "text": "There are {{ctx.payload.hits.total}} ERROR messages on Elasticsearch Cluster in the last 5 minutes.  rn One of the log entry below: rn {{ctx.payload.hits.hits.0}}"
        }
      }
    }
  }
}

Another way is GET _cat/nodes and doing a count of total nodes and alerting if value is greater than or less than the pre-defined threshold.

1