Unexpected JetStream Quorum Loss and MQTT Session Errors After Several Days of Stable Operation

[nikolaigut]

We are running NATS Server v2.12.1 with JetStream enabled and using the MQTT bridge.
Our setup includes roughly 30 MQTT clients, each publishing about 10 messages per minute with QoS 1.

After several days of stable operation, we consistently encounter the following issue sequence:

1. All MQTT clients start to fail one after another with:

   mid:74350 - "mqtt-client-id" - unable to connect: unable to persist session "mqtt-client-id" (seq=40344): wrong last sequence: 0 (10071)
   

2. Shortly afterward:

   mid:74358 - "mqtt-client-id" - unable to connect: loading session record for account "default", session "mqtt-client-id": JetStream system temporarily unavailable (10008)
   

3. Followed by repeated timeouts such as:

   mid:11409 - "mqtt-client-id" - Readloop processing time: 5.000206463s
   mid:11409 - "mqtt-client-id" - timeout after 5.000165086s: request type "MS" on "$MQTT.msgs.DEVICE.measurements" (reply="$MQTT.JSA.24jNUOPB.MS.eZtYB55V4cXzN5CODouuCa")
   

4. Finally, we observe:

   [WRN] JetStream cluster consumer 'default > $MQTT_msgs > zh49h37z_hFAM7HdYTd4NpYGiANdXUq' has NO quorum, stalled.
   

   — after which no more messages are processed.

At the same time, we notice that both CPU and memory usage spike heavily for a short period and then drop again.

The clients themselves continue to publish messages at a steady rate throughout this period.

---

Environment

• NATS version: 2.12.1
• JetStream: enabled
• MQTT Bridge: enabled
• Number of MQTT-clients: ~30
• Message rate: ~10 messages per minute per client
• Cluster setup: 3 NATS nodes
• Deployment: Kubernetes using the official nats-io Helm chart

---

Questions

1. What could cause the wrong last sequence: 0 errors during MQTT session persistence?
2. Is this related to JetStream quorum loss or a cluster synchronization problem?
3. Could high load or disk latency trigger JetStream to become temporarily unavailable?
4. Any recommended diagnostic steps or server configuration to prevent this behavior?