NH:STF S01E03 Yocto posted Wednesday, October 9, 2024 by The Neighbourhoodie Team
This post is part of our series on our work for the Sovereign Tech Fund (STF). Our introduction post explains why and how we are contributing to various Open Source projects.
You may not have heard of Yocto, but you have most certainly used Yocto, at least indirectly. Yocto lets you build a bespoke Linux distribution for your embedded devices. Many “smart home” or otherwise smart appliances likely run a variant of Linux under the hood (that you never see or interact with) and chances are good that the manufacturer used Yocto to create and maintain that bespoke Linux distribution. Their users and supporters are a veritable who is who of the technology industry: Intel, AMD, ARM, Cisco, Microsoft, Siemens, Texas Instruments, Dell, LG, Qualcomm, you name them.
—
First, we looked at security. By allowing to package any and all packages from the Linux world into a binary distribution image, Yocto immediately inherits all CVEs (Common Vulnerabilities and Exposures) of the Linux kernel itself and all of its packages. This comprises a majority of all Open Source CVEs. CVEs are the industry standard for managing security vulnerabilities in software and we are all better off for having a system that lets us manage security issues, large and small, in a structured way. However, there are a number of issues with CVEs:
- Not all CVE information is complete. Often a project releases a new version which addresses a CVE, but the original report is not updated to denote the fix version. In practice, the affected software is now listed as vulnerable forever. That’s obviously not great.
- Not all CVE information is actually correct. Sometimes affected versions and/or fix versions do not actually match reality, leading automated CVE checks to show that certain software is vulnerable when it is not. Occasionally, two software projects share the same name (by accident usually) and one’s CVEs are now attributed to the other.
- Some CVEs are so old that any information on them is difficult or impossible to find. In one case, we had to go back to a PDF published in 1993!
Finally, nobody has time to go through all CVEs and correct these shortcomings. They do not only affect Yocto but all Linux distributions and consumers of the affected software packages. Any affected project now has to maintain their own “override list” to say which CVE is not actually applicable and why. In the security compliance world, this wastes countless hours of work.
Well, nobody had time until now: we set out and in a first triage went through a whopping 221 CVEs from 1998 to 2023 and cleared them all up. We reported a sizable number of CVE corrections to NIST (The United States National Institute of Standards and Technology, which maintains the central CVE database), and we have already had a majority of our reported CVEs fixed for everybody, not just Yocto. We are extremely proud of this work.
—
The second part is no less important: usability, being a meta-Linux distribution comes with a lot of challenges that go beyond just packaging the Linux kernel and a workable userland. By allowing users to customise every part of each bespoke Linux distribution, it is imperative that any Linux distribution that can be created with Yocto is a functioning Linux distribution. To manage this immense complexity, Yocto relies on an extensive build farm and reporting from it allows any contributor and user to see where any of the desired guarantees might not hold, so they can be fixed.
However, this system has been built ad-hoc, over time and by various teams and individuals and it is not a coherent whole, so we set out to help. That’s the first part of our work: improve the tracking and visualisation of automated builds and performance tests.
In particular, we:
- Improved the performance test result pages by overhauling its graphs subsystem and design, adding graph annotations to make the data clearer, unifying styles for a cohesive project look and making it easy to get an overview of the overall project health.
- Improved the user experience, design and typography of the overgrown autobuilder index page test results and added custom filters and date range selections.
- Improved the CVE statistics view organised by release, added data annotations and inlined details to improve the overall user experience.
- Unified the multi-year and single-year project metrics views:
- Dynamically fetching chart data when the time scale changes (and speed up load times significantly).
- Connected all charts so they show the same date range, making comparisons easier.
- Unified styles with the other metrics views for a project cohesive look
- Addressed various further shortcomings with the shown project charts to make all data visualisations easier to grasp and more useful.
—
Here’s a short interview with Neighbourhoodie developers Alba Herrerías Ramírez and Jacoba Brandner, who run our STF programme and worked on Yocto with the entire rest of the team:
What was the most surprising thing working on this project?
Alba: We didn’t have to be security experts to do the CVE work and we still got a lot of CVEs corrected. Also, the Yocto infrastructure is HUGE, but documented very well. And their communication is awesome, they helped us a lot getting started.
What was especially challenging about this project?
Alba: We were making technology choices for a different team with specific requirements. Of course everything needed to be Open Source software (shout out to Apache ECharts), but also the tooling needed to be long-lasting and low-maintenance. In the end we believe we found a good set that helped the project long-term.
Yocto is using an email-based development workflow, including sending patches. Our team usually works on GitHub or other similar systems, so this was a bit of an adjustment for us.
Jacoba adds: CVE triage is a lot of somewhat tedious archival work that gets lonely very quickly. We switched to a pair “programming” model and that helped keep morale up a lot better.
What lessons did you learn on this project that will benefit the project going forward, or that you will bring into future projects?
Jacoba: We were early in establishing the whole STF process workflow and we learned here that early on, we could have saved some time by requesting help for development setups of their infrastructure from Yocto. We were initially working under the assumption that we should not make more work for the projects, but we took it a bit too far. The Yocto folks were always happy and quick to respond to our questions.
Alba: This was the biggest STF project we worked on at that time and it was great that we could find things to work on, which means you don’t have to be an expert to contribute to projects of this size. It’s a huge confidence boost for our team.
Conclusion
In summary, this was a lot of work and sometimes it wasn’t all glorious digging out patches from arcane software archives (anyone remember SourceForge?), but we are happy to have helped both Yocto and the wider Linux ecosystem.
« Back to the blog post overview