Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a script for generating import configurations from the live Ignition repositories. #144

Open
wants to merge 22 commits into
base: master
Choose a base branch
from

Conversation

nuclearsandwich
Copy link
Contributor

@nuclearsandwich nuclearsandwich commented Feb 16, 2022

This script reads data from the packages.osrfoundation.org repository and prints out a reprepro-updater import configuration.

Caveats and addendums

  • Right now it only supports the -stable repositories, and hard codes the packages in each collection (taking the info at authorship time from ignition-tooling/gazebodistro but does not support fetching the data directly from it at runtime.
  • Another limitation is that it looks only at the binary-amd64 package information so it implicitly assumes no disparities between architectures.
  • The parsing of the Debian Packages file is extremely rudimentary but presently sufficient in order to avoid dependencies outside the Python standard library.

But that's enough with the caveats, what cool things does it do?

With a builtin list of igniton collection source packages, it will find and group up all of the binary packages generated by that source and create a valid reprepro import configuration for use in the ros_bootstrap repository. This will make it much easier to keep the bootstrap repository in sync with the current releases from the packages.osrfoundation.org repository and could eventually be run out-of-tree as part of the upstream team's release automation.

#143, #150, #152, and #153 are all created with iterations on this script

@nuclearsandwich
Copy link
Contributor Author

No reviews yet but FYI @j-rivero @chapulina

@chapulina
Copy link
Contributor

Right now it only supports the -stable repositories

This should be enough for our needs 👍

@nuclearsandwich nuclearsandwich self-assigned this Apr 15, 2022
@nuclearsandwich nuclearsandwich marked this pull request as ready for review April 15, 2022 19:24
@nuclearsandwich
Copy link
Contributor Author

@j-rivero I only just learned about gazebo-tooling/gazebodistro#4 today and it looks like we thought very similarly here.

What I like from the script in that repository is that it uses the gazebodistro repos files at runtime to get the base package information, and then includes all dependencies of selected packages from the packages.osrfoundation.org repository.

What I like about this script is that it generates the exact configuration file reprepro-updater wants to consume and it can be run on any host platform rather than being dependent on the system repositories for state.

I think we should talk through the current status of each script and see which one to concentrate our efforts on.

@j-rivero
Copy link
Contributor

I'm giving a try to the tool and can not make it work with python 3.8 and 3.9.5:

❯ python3.9 generate_ignition_import_config.py 
Traceback (most recent call last):
  File "/home/jrivero/code/infra/reprepro-updater/scripts/generate_ignition_import_config.py", line 44, in <module>
    class PackageGroup:
  File "/home/jrivero/code/infra/reprepro-updater/scripts/generate_ignition_import_config.py", line 46, in PackageGroup
    packages: None|List[str] = None
TypeError: unsupported operand type(s) for |: 'NoneType' and '_GenericAlias'

Is this kind o hinting support in newer versions or there is a bug somehow?

@j-rivero I only just learned about gazebo-tooling/gazebodistro#4 today and it looks like we thought very similarly here.

What I like from the script in that repository is that it uses the gazebodistro repos files at runtime to get the base package information, and then includes all dependencies of selected packages from the packages.osrfoundation.org repository.

What I like about this script is that it generates the exact configuration file reprepro-updater wants to consume and it can be run on any host platform rather than being dependent on the system repositories for state.

I think we should talk through the current status of each script and see which one to concentrate our efforts on.

+1 Did not really like the gazebodistro PR so +1 to merge efforts here.

expected_group_version = packages_file.packages[p].version
elif packages_file.packages[p].version != expected_group_version:
print(f'{p} in {group.packages[0]} does not match expected version {expected_group_version}', file=sys.stderr)
exit(2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
exit(2)
group.packages.remove(p)

The check is nice to have. However It can detect problems with versions for orphaned packages, like the bad named libgz-math6-eigen-dev now renamed to libgz-math6-eigen3-dev. If we keep the exit after the check, the script can not continue.

A possible path forward would be to leave the warning but remove the package from the expected imports.

This was referenced Aug 25, 2022
@azeey
Copy link

azeey commented Aug 21, 2023

@azeey

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants