Erlang application stop order

An EMQ X troubleshooting case study.

Key words: emqx, shutdown crash, shutdown order, race condition

EMQ X is an open-source MQTT broker built on Erlang/OTP which can serve massive amount of TCP/TLS connections. The underlying library for listening and accepting MQTT connections is called esockd.

The trouble

Often (especially under heavy load), when shutdown the broker, we may observe error logs like below.

The only way to explain it, is that by the time when shutting down connections from esockd_connection_sup, the emqx_broker process is being shutdown (hence causing gen_server caller to EXIT with reason {shutdown, TheCall} .

To reason why such shutdown order may happen, we need to take a closer look at the emqxand esockd applications.

In EMQ X, emqx app depends on esockd app, so esockd is start prior to emqx. Shutdown is, as expected, done in the reversed order: emqx is stopped before esockd

EMQ X delegates its process supervision to esockd, in emqx_app:stop/1, it first tries to stop all esockd listeners (and supervised connection processes) like this:

All should work as expected ? No, not really, from the injected io:format logs, it looks confusing: it seems that emqx_sup is stopped before socket showdown, but the logs are mixed.

Let’s dig deeper:

In Erlang documentation

Module:stop(State)

This function is called whenever an application has stopped. It is intended to be the opposite of Module:start/2 and is to do any necessary cleaning up. The return value is ignored….

Confirm in source code: application_master.erl

Meaning, in EMQ X, emqx_sup is stopped before the listeners.

The fix

The fix is simple, make use of the application behaviour's prep_stop/1 callback.

Copyright notice: this article was originally written by EMQ. If you want to reprint, please indicate the source clearly. The link of original article:https://www.emqx.com/en/blog/erlang-application-stop-order

EMQ is an open-source IoT data infrastructure software provider, delivering the world’s leading open-source MQTT message broker and stream processing database.