Add overlays respecting steps in a pipeline and user requirements for each step

[pacospace]

Is your feature request related to a problem? Please describe.
The use of overlays directory highlights the different requirements that are used during ML project lifecycle. The management and optimization of each step can be done automatically using Thoth recommendation engine and bots. Different images corresponding to the overlays will be created automatically and used in a pipeline (e.g Elyra using Kubeflow Pipeline).

References:

• https://github.com/thoth-station/thamos#overlays-directory

cc @fridex @goern @harshad16

[tumido]

Few questions: How are these different images stored and how this maps to the build section of the .aicoe-ci.yaml file? If we want to commit to this, we need to be able to release these images to quay as well. How this mapping works? Can you point me to any docs on that, please?

[fridex]

Few questions: How are these different images stored and how this maps to the build section of the .aicoe-ci.yaml file?

Base images are stored in .thoth.yaml file as part of configuration specific to overlays (see these docs). IIRC, AICoE CI will pick this configuration. @harshad16 will know more on the CI side.

[pacospace]

Few questions: How are these different images stored and how this maps to the build section of the .aicoe-ci.yaml file? If we want to commit to this, we need to be able to release these images to quay as well. How this mapping works? Can you point me to any docs on that, please?

Hey @tumido you can find more about that here as well: https://www.youtube.com/watch?v=sQYHs58sEFg cc @harshad16

And we introduced also in jupyterlab-requirements extension for dependency management, basically every notebook will have its own kernel optimized and dependencies stored in overlays/{kernel_name} (thoth-station/jupyterlab-requirements#80) so that build pipeline will work and thoth can recommend on those stacks. I'm writing documentation for that (thoth-station/jupyterlab-requirements#96) and have a demo next monday during thoth demo session with @harshad16 :) feel free to join us! This would be combined with the work @fridex and @harshad16 did!

[harshad16]

@tumido I m in process of updating the .aicoe-ci..yaml file with all the relevant information and would update these with full details in the docs. Thanks for the question