NetBSD Accomplishes Reproducible Builds

[phoronix]

Phoronix: NetBSD Accomplishes Reproducible Builds

A lot of Linux distributions have been focusing on reproducible builds support in the past few years -- ensuring individuals can rebuild a bit-for-bit replica of the original source code. NetBSD has now accomplished their operating system can be built in a reproducible build fashion...

http://www.phoronix.com/scan.php?page=news_item&px=NetBSD-Reproducible-Builds

[tildearrow]

Sorry Azpegath if typo reporting is disturbing.

Michael Typo:

and testing on oother platforms.

[anarki2]

I find it surprising it's a thing. It should be pretty much a given

[fguerraz]

I find it surprising it's a thing. It should be pretty much a given

It's actually pretty hard to do. Being able to reproduce a build means having the exact same version for all the tools used in the build chain: linker, compiler, various helpers, etc. The package must be build with no host-dependent compiler heuristics (such as mtune). It still doesn't sound that hard but the truth is, every time something is updated in the build chain, the maintainers do not recompile every package that would take a huge amount of computing power.
So if you want reproducible builds, you need to add enough meta-data to your packages so that you can reproduce the exact environment the package was built with, and you have to maintain mirrors that contains all the possible build dependencies versions.
I'm probably missing a few things still, but it's a lot harder than it seems at first glance!

[cb88]

NetBSD guys are doing impressive stuff. With OpenBSD dropping sparc32 I know where to donate!

[waxhead]

Very nice to read about NetBSD for a change

[Azpegath]

Sorry Azpegath if typo reporting is disturbing.

Michael Typo:

No, no, no, I apologize, you misunderstood me. I find your task honorable. I do think that Michael could at least find the energy to run his text through a word processor or at least some spell checking to find the obvious errors.

[paulpach]

It's actually pretty hard to do. Being able to reproduce a build means having the exact same version for all the tools used in the build chain: linker, compiler, various helpers, etc. The package must be build with no host-dependent compiler heuristics (such as mtune). It still doesn't sound that hard but the truth is, every time something is updated in the build chain, the maintainers do not recompile every package that would take a huge amount of computing power.
So if you want reproducible builds, you need to add enough meta-data to your packages so that you can reproduce the exact environment the package was built with, and you have to maintain mirrors that contains all the possible build dependencies versions.
I'm probably missing a few things still, but it's a lot harder than it seems at first glance!

This is all true.

There is another thing that is very subtle that can break reproducible builds: the order in which files are linked.
Suppose you compile and link files like this:

gcc file1.c file2.c -o myprogram

That can produce different results than:

gcc file2.c file1.c -o myprogram

Because the linker finds the functions in different order and may place them in different parts of the binary. If a build tool is not designed for reproducible builds, it might use hash tables inside which don't guarantee order, so it might end up linking the files in different orders.

Also the compiler itself needs to be designed for reproducible results. As far as I am aware, gcc and llvm both do this.