If D-Bus goes down while stratisd is operating warn messages will flood the log

[mulkieran]

The problem is right here: https://github.com/stratis-storage/stratisd/blob/master/src/dbus_api/connection.rs#L77 . The timing of the whole loop is controlled by the wait time on the D-Bus, which is 1/10 of a second. The timing of the loop should probably be decoupled from the wait time on the D-Bus and the frequency with which stratisd checks for termination should be decoupled from the frequency with which it checks the D-Bus.

There is no obvious library based solution for this, as with the retry crate.

Basically, our goal should be to have a principled, utterly regular time to check for the down signal, and to back off checking the D-Bus if there is an error. This is especially important if the error returns immediately, i.e., does not wait 1/10 of a second.

[jbaublitz]

Just wanted to add the additional context to the issue that we've already discussed. While working on a fix for this, I determined that stopping D-Bus will also stop stratisd as a result of our service type in our systemd unit file. I believe that this problem will go away once #3927 is merged. At the very least the log will not get flooded with messages even if stratisd goes down.