Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modules lists aren't generated deterministically/reproducibly #88

Open
rpurdie opened this issue Feb 5, 2020 · 0 comments
Open

Modules lists aren't generated deterministically/reproducibly #88

rpurdie opened this issue Feb 5, 2020 · 0 comments

Comments

@rpurdie
Copy link
Contributor

rpurdie commented Feb 5, 2020 •
edited
Loading

We (the Yocto Project) are running into issues where Makefile.config and config.sh are different depending on the host or in fact the order of the modules files on disk. This leads to differences in ordering of the extensions=, known_extensions=, nonxs_ext= and dynamic_ext= lines.

One way of fixing it would be this change:

Index: perl-5.30.1/cnf/configure_mods.sh
===================================================================
--- perl-5.30.1.orig/cnf/configure_mods.sh
+++ perl-5.30.1/cnf/configure_mods.sh
@@ -82,7 +82,7 @@ extonlyif() {
 }
 
 definetrimspaces() {
-	v=`echo "$2" | sed -r -e 's/\s+/ /g' -e 's/^\s+//' -e 's/\s+$//'`
+	v=`echo "$2" | sed -r -e 's/\s+/ /g' -e 's/^\s+//' -e 's/\s+$//' | xargs -n1 | LANG=C sort | xargs`
 	define $1 "$v"
 }

the LANG=C being to ensure the sort order is deterministic and not dependent on the host locale.
halstead pushed a commit to openembedded/openembedded-core that referenced this issue Feb 7, 2020
@rpurdie
perl: Fix various reproducibile build issues
358636e 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

Signed-off-by: Richard Purdie <[email protected]>
halstead pushed a commit to openembedded/openembedded-core that referenced this issue Feb 7, 2020
@rpurdie
perl: Fix various reproducibile build issues
08f4a2f 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

Signed-off-by: Richard Purdie <[email protected]>
armcc pushed a commit to lgirdk/poky that referenced this issue Feb 7, 2020
@rpurdie
perl: Fix various reproducibile build issues
30e1e6b 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: 08f4a2f0ffeb2e8cc68f4c5ed098ec26e37cb9f8)

Signed-off-by: Richard Purdie <[email protected]>
kraj pushed a commit to YoeDistro/openembedded-core that referenced this issue Feb 7, 2020
@rpurdie @kraj
perl: Fix various reproducibile build issues
fd5611c 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

Signed-off-by: Richard Purdie <[email protected]>
halstead pushed a commit to openembedded/openembedded-core that referenced this issue Feb 7, 2020
@rpurdie
perl: Fix various reproducibile build issues
b305acf 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

Signed-off-by: Richard Purdie <[email protected]>
kraj pushed a commit to YoeDistro/poky-old that referenced this issue Feb 7, 2020
@rpurdie
perl: Fix various reproducibile build issues
bff955b 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: b305acfbd36b03774135e3d24b66004886c3b66f)

Signed-off-by: Richard Purdie <[email protected]>
kraj pushed a commit to YoeDistro/poky-old that referenced this issue Feb 8, 2020
@rpurdie
perl: Fix various reproducibile build issues
6db4f33 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: b305acfbd36b03774135e3d24b66004886c3b66f)

Signed-off-by: Richard Purdie <[email protected]>
halstead pushed a commit to openembedded/openembedded-core that referenced this issue Feb 8, 2020
@rpurdie
perl: Fix various reproducibile build issues
515bd7b 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

Signed-off-by: Richard Purdie <[email protected]>
kraj pushed a commit to YoeDistro/poky-old that referenced this issue Feb 8, 2020
@rpurdie
perl: Fix various reproducibile build issues
2bb6a37 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: 515bd7b36d6819f1b76a9c408931cfd223d05f04)

Signed-off-by: Richard Purdie <[email protected]>
halstead pushed a commit to openembedded/openembedded-core that referenced this issue Feb 8, 2020
@rpurdie
perl: Fix various reproducibile build issues
482fd0d 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

Signed-off-by: Richard Purdie <[email protected]>
kraj pushed a commit to YoeDistro/poky-old that referenced this issue Feb 8, 2020
@rpurdie
perl: Fix various reproducibile build issues
2ac1632 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: 482fd0d99f989b5a72a25bdf402fb2f219420b5d)

Signed-off-by: Richard Purdie <[email protected]>
jpuhlman pushed a commit to MontaVista-OpenSourceTechnology/poky that referenced this issue Feb 8, 2020
@rpurdie @jpuhlman
perl: Fix various reproducibile build issues
3ae8c49 
Source: poky
MR: 00000
Type: Integration
Disposition: Merged from poky
ChangeID: 2ac1632
Description:

Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: 482fd0d99f989b5a72a25bdf402fb2f219420b5d)

Signed-off-by: Richard Purdie <[email protected]>
Signed-off-by: Jeremy Puhlman <[email protected]>
halstead pushed a commit to openembedded/openembedded-core that referenced this issue Feb 8, 2020
@rpurdie
perl: Fix various reproducibile build issues
def3a9d 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: 482fd0d)

Signed-off-by: Richard Purdie <[email protected]>
kraj pushed a commit to YoeDistro/poky-old that referenced this issue Feb 8, 2020
@rpurdie
perl: Fix various reproducibile build issues
d782347 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: 482fd0d99f989b5a72a25bdf402fb2f219420b5d)

(From OE-Core rev: def3a9d748564883d71c506726554df622701b00)

Signed-off-by: Richard Purdie <[email protected]>
halstead pushed a commit to openembedded/openembedded-core that referenced this issue Feb 11, 2020
@rpurdie
perl: Fix various reproducibile build issues
1f630fe 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: 482fd0d)

(From OE-Core rev: def3a9d)

Signed-off-by: Richard Purdie <[email protected]>
Signed-off-by: Armin Kuster <[email protected]>
jpuhlman pushed a commit to MontaVista-OpenSourceTechnology/poky that referenced this issue Feb 12, 2020
@rpurdie @jpuhlman
perl: Fix various reproducibile build issues
5af360e 
Source: poky
MR: 00000
Type: Integration
Disposition: Merged from poky
ChangeID: 02cbb68
Description:

Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: 482fd0d99f989b5a72a25bdf402fb2f219420b5d)

(From OE-Core rev: def3a9d748564883d71c506726554df622701b00)

(From OE-Core rev: 1f630fe43ec3c3e78c25f93d6badc8a35ff782ad)

Signed-off-by: Richard Purdie <[email protected]>
Signed-off-by: Armin Kuster <[email protected]>
Signed-off-by: Richard Purdie <[email protected]>
Signed-off-by: Jeremy Puhlman <[email protected]>
daregit pushed a commit to daregit/yocto-combined that referenced this issue May 22, 2024
@rpurdie
src/poky:perl: Fix various reproducibile build issues
3bf077e 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: 482fd0d99f989b5a72a25bdf402fb2f219420b5d)

Signed-off-by: Richard Purdie <[email protected]>
daregit pushed a commit to daregit/yocto-combined that referenced this issue May 22, 2024
@rpurdie
src/poky:perl: Fix various reproducibile build issues
af43258 
Add a patch which handles the following issues:

a) Remove the \n from configure_attr.sh since it gets quoted differently depending on
   whether the shell is bash or dash which can cause the test result to be incorrect.
   Reported upstream: arsv/perl-cross#87

b) Sort the order of the module lists from configure_mods.sh since otherwise
   the result isn't the same leading to makefile differences.
   Reported upstream: arsv/perl-cross#88

c) Sort the Encode::Byte byte_t.fnm file output (and the makefile depends whilst
   there for good measure)
   This needs to go to upstream perl (not done)

d) Use bash for perl-cross configure since otherwise trnl gets set to "\n" with bash
   and "" with dash
   Reported upstream: arsv/perl-cross#87

(From OE-Core rev: 482fd0d99f989b5a72a25bdf402fb2f219420b5d)

Signed-off-by: Richard Purdie <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rpurdie