Application leader election
This topic describes the process of performing client-side leader elections using sessions and Consul key/value (KV) store.
Tip
The content of this guide also applies to Consul clusters hosted on HashiCorp Cloud (HCP).
Note
This tutorial is not related to Consul's leader election. If you are interested in the leader election used internally by Consul, please refer to the consensus protocol documentation instead.
Background
Some distributed application, like HDFS or ActiveMQ, require to setup one instance as a leader. This ensures the application data is current and stable.
Consul's support for sessions and watches allows you to build a client-side leader election process where clients use lock on a key in the KV datastore to ensure mutual exclusion and to gracefully handle failures.
All service instances that are participating should agree on a given key to coordinate. A good pattern is simply:
service/<service name>/leader
Requirements
- A running Consul server
- A path in the Consul KV datastore to acquire locks and to store infromation about the leader
For this guide, our key will be
service/leader
. - If ACLs are enabled, a token with
session:write
permissions over the service session namekey:write
permissions over the agreed key for this topic we expose the token using theCONSUL_HTTP_TOKEN
environment variable
- The
curl
command
Client-side leader election procedure
The workflow for building a client-side leader election process is composed of the following steps:
For each client trying to acquire the lock:
- Create a session associated with the client node.
- Try to acquire the lock on the agreed key in the KV store using the
acquire
parameter. - Watch the KV key to verify if the lock was released and if no lock is present, try to acquire a lock.
For the client that acquires the lock
- Periodically, renew the session to avoid expiration.
- Optionally, release the session.
For other services
- Watch the KV key to verify there is at least one process holding the lock.
- Use the values written under the KV path to identify the leader and update configuration accordingly.
Create a new session
Create a configuration for the session.
Create a session using the Session HTTP API.
Acquire the lock
Watch the KV key for locks
Renew a session
Release a session
$ export CONSUL_HTTP_TOKEN=cc3c73f6-667a-339b-3de1-7167f8465c68
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--data '{"Name": "test"}' \
--request PUT \
http://127.0.0.1:8500/v1/session/create
{
"ID":"edccc8f5-62a6-dcbf-045b-514b440c9bc3"
}
Check existing sessions
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--request GET \
http://127.0.0.1:8500/v1/session/list | jq
[
{
"ID": "edccc8f5-62a6-dcbf-045b-514b440c9bc3",
"Name": "test",
"Node": "hashicups-db-0",
"LockDelay": 15000000000,
"Behavior": "release",
"TTL": "",
"NodeChecks": [
"serfHealth"
],
"ServiceChecks": null,
"CreateIndex": 131,
"ModifyIndex": 131
}
]
Acquire the session
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--data '{"Node": "'`hostname`'"}' \
--request PUT \
http://localhost:8500/v1/kv/service/dbservice/leader?acquire=edccc8f5-62a6-dcbf-045b-514b440c9bc3 | jq
$ curl -X PUT -d '{"Name": "dbservice"}' http://localhost:8500/v1/session/create
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--data @svc-hashicups-db-api.json \
--request PUT \
http://127.0.0.1:8500/v1/agent/service/register