DaemonClient.is_daemon_running does not check for stale pid file

[danielhollas]

The way that we determine whether a daemon is running currently involves reading PID of its process from its PID file.

We also have a mechanism that checks whether such PID file is stale (such as when the daemon is stopped ungracefully). This is defined in function self._clean_potentially_stale_pid_file.

aiida-core/src/aiida/engine/daemon/client.py

Line 577 in 7402c17

def _clean_potentially_stale_pid_file(self) -> None:

This function is called for example when running verdi daemon stop or verdi daemon start. Notably, it is NOT called inside the DaemonClient.is_daemon_running which is used in various places around AiiDA.

I fully realize that this might not be a bug per se, and is perhaps intended due to a fragile nature of clean_potentially_stale_pid_file, but it certainly seems like this may result in inconsistent behaviour, for example when running verdi process play|pause|kill command, which could silently be failing when the daemon is not actually running.

Related issues: #5934, #5933

[sphuber]

This design decision was kind of intentional: #5940
I still feel this behavior is kind of justified because having to check whether the PID file is stale each time the status of the daemon is checked bears a non-negligible cost and this situation typically only occurs when a system is shutdown in a hard manner.