DOCS: Add to Controller section of tutorial

[chrisjsewell]

It's as clear as mud exactly the best practice for setting up and using controllers (outside of aiida-core), and would be good to add some more toy examples to the tutorial.

The test suite does not really provide much clarification in this matter and there's some open questions for me.
For example, you can launch a process by adding a ProcessLauncher to the controller, but it does not add the communicator to the created process, and so you cannot control it after launch.

Also some clarification on the interplay between kiwipy and plumpy futures, and the use of LoopCommunicator and unwrap_kiwi_future, etc

[muhrin]

Hi Chris,

yes, this isn't very clear so good that you bring it up.

ProcessLauncher and Communicator

When desining plumpy I mainly created it outside of AiiDA not particularly because I thought it would be used independently but primarily to force myself to think in terms of 'separation of concerns' and modularity. This leads to some artefacts such as what you're pointing out here that there's basically no integrated use case involving the launcher and controller together. These only come togher in the Runner in AiiDA itself and indeed, that's where most of the integration testing comes in.

LoopCommunicator and futures

This has been a source of pain and confusion for several of us and so if you (with a fresh pair of eyes) can see a way to untangle this suggestions are very welcome. We had to have a host of tornado/asyncio/concurrent.futures stuff which necessitated some unfortunate juggling of different types of futures but this is somewhat streamlined now.

So, here's the current state of play (as best I can remember): The kiwipy.Communicator interface deals in concurrent.futures.Futures because it is running a separate thread to keep communication alive. It also has a few callback hooks you can register on (add_task_subscriber, add_rpc_subscriber, etc). Both these callbacks and kiwipy.Future (which are just concurrent.futures.Futures) callbacks (via future.add_done_callback) call you from kiwipy's thread which you shouldn't block. This is where the LoopCommunicator comes it. It tries to act as a transparent shim that intercepts the callback and schedules it on the plumpy/AiiDA event loop in which case the client getting the callback can block for as long as they need. There is one more complication which is that, for example, if you do a communicator rpc_send it is legitemate for the (kiwipy) future you got back to resolve to a future. This indicates that the recipient got your message, and is working on it, but it will take some time. This is why there are a couple of recursive layers in convert_to_comm and plum_to_kiwi_future.

Sorry if this is all a little complicated but as I say, we struggled to simplify this pinch point (without breaking something else).

Hope that helps!

[chrisjsewell]

Thanks @muhrin thats really helpful.
I knew there was method in the madness lol.
Oh and obviously let me know if it looks like there is anything I have misunderstood writing https://plumpy.readthedocs.io/en/latest/tutorial.html