Uncaught error: {badrecord,not_found} / Hex get_package request failed enetunreach

[mobileoverlord]

👋 I have a bit of a complicated environment that I am trying to debug. The error I am receiving is from cross compiling the fast_tls package within the Yocto build system. It appears that rebar3 is attempting to resolve plugins, in this case pc from hex and download it for the package. For unknown reasons, it's failing to download it with enetunreach. Later in the compilation I get an Uncaught error: {badrecord,not_found}. I am assuming these to be related. The curious part is that the larger project this dependency is part of can reach the internet, and happily pulls all the other deps from hex just fine.

I am using:
rebar 3.23.0
Erlang 26.2.5
Elixir 1.16.3

DEBUG: Executing shell function do_compile
===> Getting definition for package pc from repo hexpm (#{name => <<"hexpm">>,repo_verify_origin => true,
         repo_name => <<"hexpm">>,repo_url => <<"https://repo.hex.pm">>,
         repo_organization => undefined,api_url => <<"https://hex.pm/api">>,
         repo_verify => true})
===> Hex get_package request failed: {error,
                                      {failed_connect,
                                       [{to_address,{"repo.hex.pm",443}},
                                        {inet,[inet],enetunreach}]}}
===> Failed to update package pc from repo hexpm
===> throw {error,{rebar_app_utils,{missing_package,<<"pc">>,undefined}}} [{rebar_app_utils,
                                                                            update_source,
                                                                            3,
                                                                            [{file,
                                                                              "rebar_app_utils.erl"},
                                                                             {line,
                                                                              335}]},
                                                                           {rebar_prv_install_deps,
                                                                            '-handle_profile_level/7-lc$^0/1-0-',
                                                                            3,
                                                                            [{file,
                                                                              "rebar_prv_install_deps.erl"},
                                                                             {line,
                                                                              178}]},
                                                                           {rebar_prv_install_deps,
                                                                            handle_profile_level,
                                                                            7,
                                                                            [{file,
                                                                              "rebar_prv_install_deps.erl"},
                                                                             {line,
                                                                              179}]},
                                                                           {rebar_plugins,
                                                                            handle_plugin,
                                                                            5,
                                                                            [{file,
                                                                              "rebar_plugins.erl"},
                                                                             {line,
                                                                              121}]},
                                                                           {rebar_plugins,
                                                                            '-handle_plugins/4-fun-0-',
                                                                            5,
                                                                            [{file,
                                                                              "rebar_plugins.erl"},
                                                                             {line,
                                                                              102}]},
                                                                           {lists,
                                                                            foldl,
                                                                            3,
                                                                            [{file,
                                                                              "lists.erl"},
                                                                             {line,
                                                                              1594}]},
                                                                           {rebar_plugins,
                                                                            handle_plugins,
                                                                            4,
                                                                            [{file,
                                                                              "rebar_plugins.erl"},
                                                                             {line,
                                                                              101}]},
                                                                           {lists,
                                                                            foldl,
                                                                            3,
                                                                            [{file,
                                                                              "lists.erl"},
                                                                             {line,
                                                                              1594}]}]
===> Errors loading plugin pc. Run rebar3 with DEBUG=1 set to see errors.
===> Expanded command sequence to be run: []
===> Running provider: do
===> Expanded command sequence to be run: [app_discovery,{bare,compile}]
===> Running provider: app_discovery
===> Found top-level apps: [fast_tls]
	using config: [{src_dirs,["src"]},{lib_dirs,["apps/*","lib/*","."]}]
===> Getting definition for package pc from repo hexpm
===> Hex get_package request failed: {error,
                                      {failed_connect,
                                       [{to_address,{"repo.hex.pm",443}},
                                        {inet,[inet],enetunreach}]}}
===> Failed to update package pc from repo hexpm
===> throw {error,{rebar_app_utils,{missing_package,<<"pc">>,undefined}}} [{rebar_app_utils,
                                                                            update_source,
                                                                            3,
                                                                            [{file,
                                                                              "rebar_app_utils.erl"},
                                                                             {line,
                                                                              335}]},
                                                                           {rebar_prv_install_deps,
                                                                            '-handle_profile_level/7-lc$^0/1-0-',
                                                                            3,
                                                                            [{file,
                                                                              "rebar_prv_install_deps.erl"},
                                                                             {line,
                                                                              178}]},
                                                                           {rebar_prv_install_deps,
                                                                            handle_profile_level,
                                                                            7,
                                                                            [{file,
                                                                              "rebar_prv_install_deps.erl"},
                                                                             {line,
                                                                              179}]},
                                                                           {rebar_plugins,
                                                                            handle_plugin,
                                                                            5,
                                                                            [{file,
                                                                              "rebar_plugins.erl"},
                                                                             {line,
                                                                              121}]},
                                                                           {rebar_plugins,
                                                                            '-handle_plugins/4-fun-0-',
                                                                            5,
                                                                            [{file,
                                                                              "rebar_plugins.erl"},
                                                                             {line,
                                                                              102}]},
                                                                           {lists,
                                                                            foldl,
                                                                            3,
                                                                            [{file,
                                                                              "lists.erl"},
                                                                             {line,
                                                                              1594}]},
                                                                           {rebar_plugins,
                                                                            handle_plugins,
                                                                            4,
                                                                            [{file,
                                                                              "rebar_plugins.erl"},
                                                                             {line,
                                                                              101}]},
                                                                           {lists,
                                                                            foldl,
                                                                            3,
                                                                            [{file,
                                                                              "lists.erl"},
                                                                             {line,
                                                                              1594}]}]
===> Errors loading plugin pc. Run rebar3 with DEBUG=1 set to see errors.
===> Running provider: {bare,compile}
===> Compile (untagged)
===> Running hooks for compile in app fast_tls (deps/fast_tls) with configuration:
===> Setting paths to [plugins]
===> 	{provider_hooks, [{pre, [{pc,compile}]}]}.
===> Running provider: {pc,compile}
===> Uncaught error in rebar_core. Run with DIAGNOSTIC=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: {badrecord,not_found}
===> Stack trace to the error location:
[{providers,do,2,[{file,"providers.erl"},{line,90}]},
 {rebar_core,do,2,[{file,"rebar_core.erl"},{line,155}]},
 {rebar_hooks,run_provider_hooks_,6,[{file,"rebar_hooks.erl"},{line,70}]},
 {rebar_hooks,run_all_hooks,6,[{file,"rebar_hooks.erl"},{line,18}]},
 {rebar_prv_compile,'-compile/4-lc$^0/1-0-',3,
                    [{file,"rebar_prv_compile.erl"},{line,152}]},
 {rebar_prv_compile,compile,4,[{file,"rebar_prv_compile.erl"},{line,152}]},
 {rebar_prv_compile,compile,3,[{file,"rebar_prv_compile.erl"},{line,143}]},
 {rebar_prv_bare_compile,do,1,
                         [{file,"rebar_prv_bare_compile.erl"},{line,71}]}]

[ferd]

This would point to a network being unreachable. It specifically happens when trying to contact repo.hex.pm:443, so I would have to imagine something about the environment causing issues there. Any proxy, firewall, IPv6 funky config that could be in play?

[mobileoverlord]

That's the real curious part, hex and other tools are able to use the internet just fine during the same run. It's only this isolated instance where it's receiving this error. What is special about the method in which this is being called which causes enetunreach for just this call.

[ferd]

Hex uses some 3 years old vendored code in https://github.com/erlang/rebar3/tree/main/apps/rebar/src/vendored that relies on httpc for calls. Rebar3 also uses httpc, but I imagine that re-vendoring the code on a newer version would probably help a bit.

Otherwise I'm not quite sure because it is the same underlying library, but it has to maybe inject a different configuration that fast_tls doesn't deal with the same.

[mobileoverlord]

Is there a way that I could circumvent this procedure by manually downloading pc and placing it in a required location to skip the fetch? It might be helpful in getting to the heart of the issue.

[ferd]

I've just prepared #2898 to see if newer libs work.

If you want to try vendoring as an approach, the rebar3 experimental vendor command would run in a project, vendor both plugins and libraries, and output a message telling you what config to change to have them included. This brings dependencies to the source level. If you don't do that, you could just move files into _build in the right spot and make sure the rebar.lock records match and it would work, but I would not check that into source control.