Documentation does not mention Windows binary of Skopeo.

[tandeday]

A quick read of the documentation did not immediately reveal how I got a Windows (10 if that matters) binary of skopeo for me to use against our OpenShift cluster. I do not have Go experience so I didn't just go ahead and build it.

It would be nice if the documentation clearly stated:

• If a binary is available for download (probably isn't).
• How to create one if possible? (along with a difficulty estimage).
• If not possible within reason, then explicitly mention that.

Thanks

/Thorbjørn

[vrothberg]

Hi @m86194, thanks for opening this issue.

We don't offer binaries for any OS. Skopeo is (build) tested against Linux and Mac OS but I am not sure how far we can come on Windows. I don't have access to a Windows machine for testing.

@mtrmac @rhatdan WDYT?

[mtrmac]

We do test Windows cross-compilation of c/image, so it is probably buildable, but I wouldn’t be at all surprised if Skopeo were a pretty bad Windows citizen (WRT configuration file paths and the like), and not extremely surprised if it were broken due to different path syntax or the like (though there was at least one Windows-based user of the c/image library at one point).

make binary-local would be the default way to try to build it, adding some BUILDTAGS might be necessary.

Frankly, making Windows work, and work smoothly, and having everything well documented, would be mostly dependent on someone interested contributing that. We are definitely interested, at least enough not to sabotage that :) , but lack both access and time to do it ourselves.

Same for providing pre-built binaries; on Linux it is more or less an intentional choice not to, because a single binary is not portable to various Linuxes; on Windows it might be more practical, but again dependent on someone integrating that into the current build system.

[xphyr]

SO, it very much seems possible. I was able to build a Windows binary just this evening (completely unrelated to @m86194 request). I made a small tweak to the Makefile (created a Windows build target) and it does compile. I will test it out tomorrow at work on a Windows machine and see if its usable. If you are curious about the addition to the Makefile take a look at my fork of this project.

[tandeday]

First of all, please notice I am not requesting a windows build (which would be nice, though) but that the documentation is clearer about what can be expected from a new user coming to the project. (I need a dockerless image mirrorer) because it is slowly showing up in google searches for this kind of utility.

I fully understand that you have limited resources and this is to be considered an internal Redhat tool which just happens to be useful to others, instead of a fully funded product. I am just letting you know that people like me want to know this pretty quickly when triaging posibillities.

@xphyr this sounds very interesting. If it is usable, I'd like to have a look at it. Thanks.

[rhatdan]

Well, first of all this is not an Internal Red Hat Tool. Skopeo is a fully open source project and we welcome it running on any platform. Linux, MAC, Windows ...

If we get contributors working on the Windoews and MAC Ports that would be great. The primary developers right now are Linux users and therefore it gets the most attention.

We are working to make Podman easier to consume on Windows and MAC, so I don't see a reason why we would not want to get Skopeo easily available on those platforms as well.

[xphyr]

I was able to test the windows binary and it works for what we needed (doing remote inspection of images). From an initial look, I believe the Mac Port and Windows port would be similar in nature, leveraging the "binary-static" targets in the Makefile to eliminate anything that requires CGO or linux only features like brtfs. I will take some additional time over the next few days to try and test the other functionality of the tool on these non-linux OSes.

[rhatdan]

Great, thanks for looking into this.

[xphyr]

Over the weekend I tested the functionality of the Windows and OSX builds, using the Make file in my fork and I documented the results in the readme as well. Generally speaking, inspect, copy & delete work, as does the various storage subsystems except those specifically tied to the linux operating system (such as "ostree support"). It looks like this was known for OSX, but not for Windows.

If there is interest, I would be happy to work on a binary release process for OSX/Windows so that others could leverage this tool on alternative platforms without the need for installing Go and Make. My first thought would be to either use either:

• goreleaser as part of the Travis CI build process
• the native TravisCI release module.

Please let me know if this is something you would like to accept into the project and I will work on putting it together.

[rhatdan]

Yes, I think we are very interested in this.

[vrothberg]

Absolutely, thanks for looking into this, @xphyr !

[xphyr]

I have been able to make relatively few changes to the build process and I now have a travis build that runs against Linux, OSX and Windows and passes all tests successfully. I am now working on how to get travis to create a github "release" so that we can upload a compiled binary for windows and osx. The on-going work is in my fork if anyone is interested in seeing progress or has any feedback.

[mtrmac]

Great!

Some quick observations looking at that branch (without checking to verify that what I’m talking about is practical right now, it may just be my ignorance):

• If possible, I’d like to minimize proliferation of build targets, and rule duplication; e.g. the primary make binary-local target, when run on macOS/Windows, should automatically produce a working macOS/Windows binary, and we shouldn’t need to tell the users to run different commands on different systems. (Your work already moves in this direction, with the OS/name conditionals). cross-OS build targets may still be useful, but they are distinctly secondary to native builds, and ideally should reuse, not duplicate, the primary build rules (e.g. a binary-local-windows target could just be make binary-local OS=Windows_NT or something like that).
• We’re, by default, emphasizing the non-container builds (make binary-local); if they work, and they are the only ones used by CI, I’m not sure we even need the container ones to be added; they bring little value and are somewhat likely to be broken without anyone noticing.
• At least on Linux, the default is to use dynamic, not static, linking, unless there is a good reason (dynamic linking is preferred to allow independent updates of the called libraries, including security fixes). I’m not sure whether this applies to macOS/Windows as well (I don’t know all that much about them) – it may well not be the same – but I’d like this to be, eventually, at least explicitly discussed.

[xphyr]

Thanks @mtrmac for the feedback. I agree the multiple build targets was getting a bit much, and I really like your solution. I will look to incorporate this into the work I am doing.
I will need to look more into static vs dynamic linking on macOS/Windows. I will admit ignorance here and say that I have only ever built static binaries for these OSes. One of the things that has made building the Windows binary more challenging is the CGO requirement for gpgme in the containers/image library. I may look into this next, just to see if there is a solution to this problem as my next project to tackle ...

[xphyr]

End of the weekend update, my football team lost, but I was able to clean up the multiple build targets, and make "make" do something much more sane. I am now detecting the need for "containers_image_openpgp" dynamically, similar to the detection of OSTREE, LIBDM and BTRFS tags. This has us back to just "binary-local" and "binary-local-static" targets. I have also backed out the explicit requirement to build static and the BUILD_OS is auto-detected on start, but can be over-ridden for testing purposes.

I also had success in publishing a Windows and OSX build binary to my fork. I have done a preliminary test of the Windows binary and it works. If you check my fork under releases you will see a Windows and OSX (darwin) release file for version 0.1.40-xphyr.

I am wondering if I should tackle #580 while I am in the Makefile making changes. After reading 580, as well as the comments above about not needing containers should I attempt to strip the container build targets from the Makefile, or do we want to do that as a separate PR (if at all).

[sebastianwebber]

hello all!

do you guys still have an interest to use GoReleaser to make the releases? Can I open a PR?

[TomSweeneyRedHat]

@sebastianwebber we're interested, @xphyr have you made any further progress?

[sebastianwebber]

hey @TomSweeneyRedHat and @xphyr I added a draft PR but i need some info to finish.

[xphyr]

Hello all, sorry I must be honest, I completely forgot about this. We got a version working for windows for my prior employer using my fork and then I had to move on... my fork is very outdated so I should really go back and take a look at if my make file changes would still work. Using GoReleaser is a good idea and I could take a look at what what @sebastianwebber has put together for it ... the question becomes does the skopeo project want to take on a different build process or stick with "make"... (BTW I am now a Red Hatter (yeah!) so I can hopefully see this through to completion now. )

I think what I need to know is if we want to incorportate fixes for #580 as well as getting a windows build working and getting binary releases out there or not.

[TomSweeneyRedHat]

@xphyr welcome to Red Hat! @mtrmac and @vrothberg 's call, but I'd like to stick with make if we can. But if it turns out it's not possible to pull off a windows build that way, I'd not be against a separate build.

@xphyr did you want to tackle #580? If not, we probably can twist @parkervanroy 's arm and ask him to take a look.

[TomSweeneyRedHat]

and I'll try using Parker's real GitHub handle this time, @ParkerVR. Parker please see the above comment.

[ParkerVR]

@TomSweeneyRedHat happy to look into this, but I don’t have much bandwidth this week and am moving next week- can get into it after that.

[xphyr]

@TomSweeneyRedHat thanks for the welcome .. who knew a year ago I would be here! I will take a look at both the windows build as well as tackling #580 I have not looked at this in a while, so will take me a few minutes to get back up to speed. I will status this issue later in the week with where I have gotten.

[rhatdan]

@ParkerVR Stick to the shortnames work.

[rhatdan]

@ashley-cui PTAL

[gbraad]

Related: containers/podman#11809

[gbraad]

@Rickaro

Can this be a batch file or even perhaps an appveyor build definition?

[tomcruise81]

Does it make any sense to consider moving from the mtrmac/gpgme gpgme wrapper that requires additional native libraries and CGO and instead moving to a native Go implementation?

From golang/go#44226:

If you're looking for alternatives, consider the crypto/ed25519 package for simple signatures, golang.org/x/mod/sumdb/note for inline signatures, Tink for various cryptographic tools, or filippo.io/age for encryption (the latter two are not maintained by the Go team). You can read a summary of OpenPGP issues and alternatives here.

If you are required to interoperate with OpenPGP systems and need a maintained package, we suggest considering a maintained community fork of golang.org/x/crypto/openpgp. We don't endorse any specific one.

So possibly:

• https://pkg.go.dev/golang.org/x/crypto/openpgp - deprecated
• https://github.com/ProtonMail/gopenpgp
• https://github.com/ProtonMail/go-crypto

It'd allow getting away from things like https://github.com/containers/skopeo/blob/4fda005a3e83727c93caa63f014d165ea1fe6cc2/cmd/skopeo/cgo_pthread_ordering_workaround.go

Are there any other aspects that rely on CGO?

[Rickaro]

@gbraad

for building 64bit version

appveyor.yml

version: 1.0.{build}
image: Visual Studio 2015
install:
- ps: |
    $path = [Environment]::GetEnvironmentVariable("path", "machine")
    $newPath = ($path.Split(';') | Where-Object { $_ -eq 'C:\ProgramData\chocolatey\bin' }) -join ';'
    [Environment]::SetEnvironmentVariable("path", $newPath, "machine")
    choco install gpg4win --no-progress
    [Environment]::SetEnvironmentVariable("path", $path, "machine")
build_script:
# APPVEYOR_BUILD_FOLDER=/c/projects/skopeo
# Here we need edit Makefile and comment line 11, starts with GPGME_ENV, in order to be able to set CGO variables
- sed -i -e '11s/^GPGME_ENV/#GPGME_ENV/' Makefile

- |-
  set GPG_INSTALL_DIR=C:\Program Files (x86)\Gpg4win
  set GPG_INSTALL_DLL=%GPG_INSTALL_DIR%\bin_64
  set DIST_DIR=dist
  set GPG_DLL=gpg-dll

- ln -s "%GPG_INSTALL_DIR%" ./
#Next - we will use dll files for linking
#x86_64 located at /c/projects/gpg4win/bin_64
#dll names should be changed for ld: libassuan6-0.dll -> libassuan.dll 
#Create symlinks for libraries, so linker can found them
     
- ps: |
    New-Item -Path $env:DIST_DIR -ItemType Directory
    New-Item -Path $env:GPG_DLL -ItemType Directory

    Get-ChildItem $env:GPG_INSTALL_DIR\bin_64 | `
      Where-Object { $_.Name -match '\d?-[^-]+?(\.dll)$' } | `
      % {
        Copy-Item -Path $env:GPG_INSTALL_DLL\$_ -Destination $env:DIST_DIR
        
        Copy-Item -Path $env:GPG_INSTALL_DLL\$_ -Destination $env:GPG_DLL -PassThru | `
          Rename-Item -NewName { $_.Name -replace '\d?-[^-]+?(\.dll)$','.dll' }
      }

    Copy-Item $env:GPG_INSTALL_DLL\gpgme-w32spawn.exe $env:DIST_DIR

- |-
    set PATH=C:\mingw-w64\x86_64-7.3.0-posix-seh-rt_v5-rev0\mingw64\bin;%PATH%
    set CGO_ENABLED=1
    set GOARCH=amd64
    set MSYSTEM_CHOST=x86_64-w64-mingw32
    set PROJ_DIR=/c/projects/%APPVEYOR_PROJECT_SLUG%
    set CC=%MSYSTEM_CHOST%-gcc 
    set CXX=%MSYSTEM_CHOST%-g++ 
    set CGO_CFLAGS=-I%PROJ_DIR%/Gpg4win/include
    set CGO_LDFLAGS=-L%PROJ_DIR%/%GPG_DLL%
    mingw32-make.exe bin/skopeo
    copy bin\skopeo %DIST_DIR%\skopeo.exe

artifacts:
  - path: dist

[gbraad]

Are there any other aspects that rely on CGO?

@lsm5 ^^

Also, it might help to have community builds happening on appveyor to allow for testing Windows, but non-blocking/unreleased. WDYT?

[mtrmac]

Does it make any sense to consider moving from the mtrmac/gpgme gpgme wrapper that requires additional native libraries and CGO and instead moving to a native Go implementation?

From golang/go#44226:

If you're looking for alternatives, consider the crypto/ed25519 package for simple signatures, golang.org/x/mod/sumdb/note for inline signatures, Tink for various cryptographic tools, or filippo.io/age for encryption (the latter two are not maintained by the Go team). You can read a summary of OpenPGP issues and alternatives here.

If you are required to interoperate with OpenPGP systems and need a maintained package, we suggest considering a maintained community fork of golang.org/x/crypto/openpgp. We don't endorse any specific one.

Yeah, so using the maintained GpgME library to interact with a maintained /usr/bin/gnupg seems better than at least that one library above.

As for other Go implementations: It’s a code that could be integrated, but a significant subset (by no means all) users that strongly need signatures are also subject to various certification requirements like FIPS 140, which requires clearing of all memory containing private keys after it is no longer used, which is pretty much impossible to do purely in a garbage-collected language like Go, without specific language/compiler/runtime support for that, or without farming the key management to a different-language implementation.

I’d guess (without any data at all) that there are quite a lot of Windows users who want Skopeo for the ordinary copy functionality, not specifically for signing, so those could just use the verification-only non-go build. Is anyone here specifically interested in signing?

Are there any other aspects that rely on CGO?

containers/storage device-mapper support (probably irrelevant for Windows), the ostree transport (probably irrelevant), PKCS#11 support for encrypted container images (probably relevant on Windows, but also somewhat niche).

[mtrmac]

Also, it might help to have community builds happening on appveyor to allow for testing Windows, but non-blocking/unreleased. WDYT?

I expect it would be a rough experience, because none of the code has been written with any effort to be a good Windows citizen (e.g. use the right locations to store configuration/caches/state), and probably without much knowledge what that even means.

With that caveat, if anyone is wiling to test the current state and propose improvements, any help with Windows support would be welcome.

[bofei222]

@xphyr have you made any further progress? could i download a binary file for windows now ?

[gaborcsardi]

I needed a Windows binary on GitHub, so I used the excellent instructions at #715 (comment) to build one, and put it here: https://github.com/gaborcsardi/skopeo/releases

It is 64 bit with gpgme. It is just a zip-ped up skopeo directory that contains the required exe and dll files. The only difference to the instructions is that I did strip -x on the exe, to make it (much) smaller.

I did some very basic testing only. i.e. running a couple of commands against a remote registry.

I did run a quick Windows 10 virus scan on it, but the usual disclaimers apply. It is probably better to run the build on your machine for yourself.

[mpils]

Thanks a lot @gaborcsardi for provided skopeo for windows release!
Here are some issues i was running into using skopeo in my cmd shell and related solution:

1. you have to use
   --override-os linux [mandatory]
   --override-arch amd64 [optional]
   because skopeo use local os information overwise (os:windows) and doesn't find a container image supported for windows

.\skopeo.exe --insecure-policy --override-os linux inspect docker://registry.fedoraproject.org/fedora

for additional information

• you can find Golang $GOOS and $GOARCH supported values here
• you can inspect container image supported values using skopeo (see example above)

2. you have to use
   --insecure-policy [mandatory]
   because trust policy file /etc/containers/policy.json doesn't exist!

.\skopeo --insecure-policy --override-os linux copy --additional-tag fedora:latest docker://registry.fedoraproject.org/fedora:latest docker-archive:fedora-latest.tar

I hope somebody will find this usefull :)

[gaborcsardi]

@mpils Thanks!

1. I didn't notice this, because my use case needs skopeo copy --all.
2. An alternative is to use the --policy option to refer to a policy file explicitly. An example policy file:

     {
        "default": [
          {
            "type": "insecureAcceptAnything"
          }
        ],
        "transports":
          {
            "docker-daemon":
              {
                "": [{"type":"insecureAcceptAnything"}]
              }
          }
      }

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[PeeZu]

Thanks @gaborcsardi for the Windows version, this save my life 👍 .

Any chance to have the latest Skopeo version Windows build on your fork ?

[gaborcsardi]

@PeeZu I just put v1.9.3 at https://github.com/gaborcsardi/skopeo/releases

However, this is even less tested than the previous version, so please do let me know if it works for you or not. Thanks!

[PeeZu]

Awesome @gaborcsardi !

First usage seems working as expected, thanks !

[gbraad]

@gaborcsardi So you manually performed these steps? As commented by Rickaro, it might be helpful to set up a Powershell script or a CI defintion: #715 (comment) and contribute this to the project. Even though they might not publish releases, it does show the build artifacts and build state.

---

I am testing with an automated build as provided by @Rickaro, though some changes are needed as Skopeo needs a more recent version of the Go toolset (otherwise it fails on io/fs which got included in Go since 1.16 IIRC)

[esemplare]

I've recently come across spokeo while needing to work with images from hosts without docker runtimes to pull. This is a great utility and part of this requirement is running the tool from a windows host.

I've successfully cross compiled from the main branch following @Rickaro 's instructions here: #715 (comment) with gpgme.

I'm only using it for the aspect of copy and sync, however I'm not sure what other tests to run to ensure there are issues with the other commands.

With the change to stop calling gpgme-config and using CGO overrides (ebc5573) the build failed due to a windows install of gpg4me not including a pkg-config file. I ended up creating one manually.

Once the project build, the other issue was that it would try to create invalid windows directories, in this case failing on a colon.

We could replace with something like "_" which is what curl does, or perhaps URL encoding (%3A) so the information of what was replaced is retained.

I ended up placing this in a runtime check for windows. I didn't spend too much time on coming up with an ideal solution but it worked for now.

case directory.Transport.Name():
    if runtime.GOOS == "windows" {
        destination = strings.Replace(destination, ":", "%3A", -1)
    }

---

All this said, I have some questions:

• @gbraad , do you know what issues I should expect regarding io/fs if any? I used Go 1.20.4 windows/amd64.
• What are the thoughts on including platform specific code? Have a sync_windows.go ?
• Thoughts on the override-os comment @mpils mentioned? Should it default to linux regardless of the actual os?
• Locations for the files under /etc/containers ? Couldn't tell how those default locations are referenced.

[gbraad]

We did a small test with skopeo on Windows, but eventually decided to use different logic to deal with our requirement. An additional binary, built and maintained by us, was not the ideal path forward. So, unfortunately I can not say more than 'it worked', though our testing has been minimal. We might revisit this at some later time, though for now we have concluded and fulfilled our requirement differently.

Note: I work on CRC/OpenShift Local (and work on Podman Desktop/podman machine integrations) providing support on multiple platforms.

[mtrmac]

• What are the thoughts on including platform specific code? Have a sync_windows.go ?

Yes, typically. As an alternative, there is some precedent in c/image for using a run-time check based on comparing runtime.GOOS, which is more costly but has the benefit that the code can have unit tests that run on Linux as well… but that would only make a difference if that code had unit tests (which is not the case for most of skopeo sync now).

• Thoughts on the override-os comment @mpils mentioned? Should it default to linux regardless of the actual os?

Long-term, I’m not much of a fan — it would make it much harder to start using containers (or OCI artifacts) for non-Linux systems in the future. This equally, or perhaps even more, applies to macOS.

I appreciate that in a short-term horizon, defaulting the OS of images to copy to Linux is quite convenient. Maybe that can be achieved using a shell alias.

• Locations for the files under /etc/containers ? Couldn't tell how those default locations are referenced.

#787 contains some pointers at the implementation mechanism.

Those /etc paths are clearly not the right defaults on Windows — are there good defaults that can be used? I’m not sure Windows has an equivalent of a system-wide multi-user /etc like this, but I know very little about Windows. If there are good defaults, ideally we should add them to the containers/image codebase directly, so that they are used automatically.

[dhirschfeld]

I’m not sure Windows has an equivalent of a system-wide multi-user /etc

That's probably C:/ProgramData

[passcod]

I've made a github repo that cross-compiles the non-gpg version, and puts it up in a release. That's sufficient for my needs, but would welcome a PR to build a better version. Does so once a week so it will always keep up to date with releases. https://github.com/passcod/winskopeo

[hurzelpurzel]

Thanks to @passcod create a PR Update install documentation #2124 to refer to your packackge repository