* How do we do this today? We don't really have great tools to do this. There is monitoring, we can do static analysis etc., but these are not a direct way of ensuring our software wasn't tampered, but rather monitor the environment, and hope we catch things using static analysis etc. * This talk is about how we can practically verify integrity of software

* We specialize in supply chain security, operating system engineering, infrastructure hardening, and applied cryptography * We exclusively use and write open source software * Introduce some problems teams maybe weren't even thinking about

* Ken Thompson is a computer scientist from Bell Labs, read a Air Force paper where he got this idea * Even if you review your source code and verify it's secure, that's not enough, as the compiler can still modify code * How can you trust a compiler... but also how can you trust all the software downstream from the compiler, that's built by it. And again, how do we easily check if the compiler, or some other aspect of the environment injected malicious software (guest software)? * This is an unexplored attack surface area I will do my best to contextualize it and give you a good intuition about it * I won't open the can of worms on whether it's better to use open source software in the context of security, but I'm firmly in the camp of don't trust, verify

* One of the most significant breaches in recent history - Orion software platform - a monitoring tool to help orgs manage their infra including networks, servers, applications, dbs etc. * 1000s of enterprise and government customers had their systems completely exposed * This company is one of the GO TO companies for cybersecurity solutions * The other thing that happened is that the APT stole cybertooling and weaponized it and used to improve their evasive abilities * This means that IP, government secrets etc could have been leaked * I never saw a proper response and retro on how to prevent this from happening * Not directly the result of compiler compromise, but no way to verify if software is tampered

* We use integrity hashes to ensure that the software is not modified between the download source (CDN etc.) and end user * You may be thinking that it's likely that most software is already deterministic by default - but it's not. This is because of things like time stamps, linking order, compilation flags, environment variables etc. * This becomes very powerful when we start to reproduce the same software in multiple different environments, and by different agents. Different hardware, different OS, different person etc. * So determinism is the method that allows us to easily and quickly check if something new has been added to a binary * How do we apply this to our tech stack?

* In this example, we see the same software built deterministically in several different environments. * Because it's determnistic, we know that we expect the same hash on all systems * We easily notice that Azure produced a binary that hashes to a different value, and therefore know something is different about this binary

* We need everything to be deterministic - this is not how software is currently built * And yes this is not simple to do... so let's talk about how we can achieve this * As a side note, a similar apporach applies to interpreted languages like javascript, where can hash the source code of the application, and use a deterministically built environment and runtime.

* Maintainers of open source software are the people that often are the ones building this software, and even in large organizations like Microsoft and Apple, they are not using determinism * We can also rely on reverse engineering but it's not a reliable and practical method * So the very foundation of how we build software is not verifiable... that's a problem

* Complicated but auditable process * We want to do this deterministically of course so we have a tamper evidence method

Now this seems like a lot... and it is, so we went ahead and built an open source solution that tries to address the problem

Speaker notes * Most Linux distributions are built for *compatibility* rather than *security*. This results in a dramatic increase of attack surface area of an operating system. * StageX is the first Linux multisig distribution, is one of two fully bootstrapped Linux distributions, is 100% reproducible and deterministic, and can build complicated software with as few dependencies exposed as possible. * The other thing that differentiates StageX from other solutions like NixOS is that it is fully container native, so there is no package manager required such as flake or otherwise.

NOTE: "Rust deps" is the amount of dependencies required to build a Rust hello world

* This may look very similar to what you may do with alpine linux, but the difference is that with alpine you are trusting single points of failure since none of the alpine packages are multi reproduced and signed - this is why we made stagex - they also do not use bootstrapped compilers. * Who built alpine rust; what compiler did they use * There is no way to easily reproduce most software so you can't verify it for yourself, you are blindly trusting that the binary is clean

Speaker notes To ensure StageX remains a secure toolchain, there's some additional maintenance that is performed compared to most distributions. This includes: * Built using hash-locked sources. This ensures every developer gets the exact same copy of the code for each container, so no middleman could inject malware, which helps with: * Reproducing projects, ensuring they're built deterministically. This confirms that no single developer, nor their machine, have been compromised. Once each package is confirmed, they are... * Signed by the release maintainers. These maintainers each build a copy of the package locally and sign the containers with an OCI-compliant signature using well-known OpenPGP keys. ---

flowchart TB Build1[Build] --\> Reproduce1[Reproduce] Build2[Build] --\> Reproduce2[Reproduce] Reproduce1 --\> Sign1[Sign] Reproduce2 --\> Sign2[Sign] Sign1 --\> Release Sign2 --\> Release { "theme": "light", "themeCSS": ".label foreignObject { overflow: visible; }" }

TODO: talk about bootstrapping, incl. corrupt compilers in distro toolchain

https://distrowatch.com/images/other/distro-family-tree.png

Speaker notes * We have multiple individuals rebuild the all of the software in the StageX distribution * You can also clone the stagex repository, install docker and run the command `make` to verify for yourself that all the hashes match * You can overlay rules around how many times software has to have been rebuilt, and a trusted list of cryptographic keys the software has to be signed by to ensure you always have a desired level of reproduciblity in your stack ---

flowchart TD MA[Maintainer A] --\> Go MB[Maintainer B] --\> Go MC[Maintainer C] --\> Go MA --\> GCC MB --\> GCC MC --\> GCC MA --\> Rust MB --\> Rust MC --\> Rust { "theme": "light", "themeCSS": ".label foreignObject { overflow: visible; }" }

* We already offer packages that can be used today, and are used in production by multiple companies * Adding a usability improvement where all the dependencies are grouped into what we are calling "pallets"

Other distributions run their own package manager inside of containers We use containers as our package manager 100% container native, no attack surface * Package managers are notorious for introducing attack surfaces, such as arbitrary execution of `setup.py` or post-download scripts, and by using Docker as our package manager, we avoid all forms of spontaneous execution. All StageX software is built deterministically, meaning you can be sure all components listed in your Software Bill Of Materials hasn't been tampered with. Because StageX provides a toolchain for you to build your software in the same manner * Available on docker hub