Restructure and clean up of make.py

[pdeiml]

'make.py all' is doing a lot of things which need a bit clean up and restructuring.
I start with a description of the new structure followed by a description of necessary clean up later on.

The following list describes the future cli-commands of make.py and a description of them (make.py all' should execute them chronologically):

1) check-input
Basically the same as 'make.py checks --step input'. The info.yaml files and their validation should be added to the InputData class in a clean way.

2) input-index-file
Basically the same as 'make.py collection --step input-index' which creates the index file of all input data. See #175 for more details about that file.

3) catalog
Basically the same as 'make.py catalog' which produces the catalog fits-, yaml- and ecsv-file

4) check-catalog
Basically the same as 'make.py checks --step catalog' which validates the three catalog files from step 3).

5) collection
Basically the same as 'make.py collection --step [sed, lightcurve, lc]' which copies all data from the input folder to $GAMMACAT/docs/data.

6) check-collection
Basically the same as 'make.py checks --step collection' which validates the data files of the collection which are produced in step 5).

7) collection-index-file
Basically the same as 'make.py collection --step output-index' which creates the index file of all data in $GAMMACAT/docs/data/data.

7) webpage
Basically 'make.py webpage' which is under development. See

One can think about joining some of the upper steps but this can be done very easily at the end.

Moreover, there has to be done a clean up in the procedure of these steps.
In easy words: The start of step 3) and 5) should be the input-index file and not a scan through the corresponding folder.

[cdeil]

How about adding the files from input/sources to the input index file as well (and then also to the output and output index)? That data is also needed to make the catalog, and if all data needed is available via the index files the scripts and processing will become simpler. I think I would start with that, and only then move on to rewrite the data processing code to always start with the resource index.

[pdeiml]

Do you want to copy the input/sources files to docs/data as well?

[cdeil]

Do you want to copy the input/sources files to docs/data as well?

Yes. It should be scripted in the same way as SED and LC: there should be a class that processes it. It can start out mostly empty, i.e. making a copy, but then over time we'll add scripted fixes and additions before writing out the file, to make the output as uniform as possible.

[pdeiml]

After working a little bit on this IMO one should change point 1) and point 2) in the upper procedure. Because if there is already a input index file it will be easy to scan through its entries and check them against the schemas.

Hence, the new procedure is:

1) input-index-file
Basically the same as 'make.py collection --step input-index' which creates the index file of all input data. See #175 for more details about that file.

2) check-input
Basically the same as 'make.py checks --step input'. The info.yaml files and their validation should be added to the InputData class in a clean way.

3) catalog
Basically the same as 'make.py catalog' which produces the catalog fits-, yaml- and ecsv-file

4) check-catalog
Basically the same as 'make.py checks --step catalog' which validates the three catalog files from step 3).

5) collection
Basically the same as 'make.py collection --step [sed, lightcurve, lc]' which copies all data from the input folder to $GAMMACAT/docs/data.

6) check-collection
Basically the same as 'make.py checks --step collection' which validates the data files of the collection which are produced in step 5).

7) collection-index-file
Basically the same as 'make.py collection --step output-index' which creates the index file of all data in $GAMMACAT/docs/data/data.

8) webpage
Basically 'make.py webpage' which is under development.

[pdeiml]

First PRs handling point 1) and 2):

#211
#234