Implementing Secure Software Upgrades in Embedded Systems: Best Practices and TPM Integration

Timestamps:

• 0:09 – Introduction
• 3:47 – Overview of Secure Software Upgrades
• 4:35 – Secure Embedded Systems and Secure Boot
• 5:41 – Importance of Software Upgrades
• 7:22 – High-Level Characteristics of Secure Software Upgrades
• 7:32 – Trusting Firmware During Software Upgrades
• 8:30 – TPM and Platform Configuration Registers (PCRs)
• 9:09 – TPM as a Secure Key Storage Mechanism
• 9:36 – Sealing Secrets in the TPM
• 10:00 – Authorized Policy in Software Upgrades
• 10:58 – Securing the Software Upgrade Process
• 12:38 – Firmware Upgrade Latch in Secure Systems
• 13:50 – Downgrade Protection in Software Upgrades
• 14:47 – Disabling Insecure Recovery Methods
• 15:22 – Common Recovery Mechanisms
• 15:51 – A/B Upgrade Mechanisms
• 16:17 – Advantages and Trade-Offs of A/B Upgrade Mechanisms
• 17:20 – Software Update Frameworks
• 17:58 – Encrypted File Systems in Secure Systems
• 18:59 – Demonstration of TPM in Action
• 19:48 – Sealing a Secret in the TPM
• 20:17 – Unsealing the Secret
• 21:30 – Modifying PCR Values
• 22:35 – PCR Changes During Software Upgrades
• 24:10 – Reverting PCR Values
• 25:00 – Demonstrating TPM Usefulness
• 26:02 – Conclusion and Q&A

****

0:09 – Introduction

Alecia:
Welcome, everybody, to today’s webinar on implementing secure software upgrades in embedded systems, focusing on best practices and TPM integration. We’re thrilled to have you here with us as we dive deep into one of the most critical aspects of embedded systems security—ensuring its integrity and the security of software upgrades.

0:34 – Housekeeping Items

Before we begin, let’s go over some quick housekeeping items to ensure a smooth experience. Throughout today’s webinar, we will have a live Q&A session at the end of the presentation, so I do encourage you to submit your questions, and we will be answering them at the end. Feel free to type them in as they arise. And should you encounter any technical issues or need any assistance, just send a message through the chat, and our webinar admin team is here and ready to help you.

1:05 – About Fidus Systems

So, who is Fidus? Well, we were founded in 2001 and have since grown into a 150-person, all-North-American embedded system services company. We serve all industries and markets and have three brick-and-mortar locations in North America—two in Canada and one in Silicon Valley. We’ve been working with embedded systems since 2002, and we’re proud of our repeat customer rate—close to 95% of our customers return year over year with new projects and new business, a real testament to the quality and efficiency of our work and our commitment to doing it right the first time. We have an exceptionally strong partner ecosystem, with strong partnerships with AMD, Altera (formerly Intel), Analog Devices, and Lattice, to name a few.

1:53 – Partner of the Year Announcement

Most recently, this past July, we were thrilled to have been announced as Partner of the Year by AMD. In our first decade, Fidus did a ton of varied AMD Xilinx-related work, and so in 2011, Xilinx asked Fidus to become their first-ever premier design services member in North America. To this day, we have collaborated closely on opportunities and trained together on the latest technologies, such as Versal and their latest Embedded Plus architectural series—all with the end goal of helping our joint clients get their solutions to market as accurately, quickly, and smoothly as possible.

2:35 – Fidus’ Full-Service Offering

As a full-service electronic systems design firm, our professionals and engineers cover a multitude of service disciplines, including FPGA and ASIC design, embedded software, hardware, signal and power integrity, verification, and many more. We help our customers by supplying some of these expert skill sets directly or by managing the complete project from start to finish.

3:02 – Introduction of Speaker

And now, I’m pleased to introduce our speaker today, Dawson Theroux, a seasoned embedded software designer here at Fidus with deep expertise in implementing secure boot and Trusted Platform Module (TPM) integration in embedded systems. With years of experience working across various embedded platforms, Dawson has helped our customers enhance their security posture by protecting firmware and managing encryption keys during software upgrades. His extensive knowledge of TPM, secure boot processes, and real-world implementations makes him the perfect expert to guide us through today’s session on securing software upgrades. So, without further ado, I’ll hand it over to Dawson to kick off today’s session.

3:47 – Overview of Secure Software Upgrades

Dawson Theroux:
Thanks, Alecia. Hi everyone, I’m Dawson Theroux, an embedded software designer here at Fidus Systems. This webinar will explore the importance, requirements, and implementation of secure software upgrades in embedded systems. Throughout the webinar, the security features and precautions of secure software upgrade mechanisms will be explored. This includes details regarding overall system security in embedded systems, the reasoning and implementation details behind secure software upgrades. Also covered is the use of TPMs to store secrets, such as authentication and encryption keys used during the upgrade process, and common recovery mechanisms that can be implemented to recover a secure system after a failed upgrade.

Let’s get started by going over the common features of secure embedded systems.

4:35 – Secure Embedded Systems and Secure Boot

A secure embedded system consists of many components and configurations to offer system-wide security. Secure Boot is the first component. It establishes the Root of Trust for the system by offering authentication, integrity, and encryption to bootloaders. For more information about Secure Boot, take a look at our previous webinar on the subject.

Once Secure Boot establishes the Root of Trust for the bootloaders, the boot process is started, and the bootloaders must authenticate later components in the boot process to extend the Root of Trust. In addition to system boot security, it is important to lock down system access through system configuration. Some examples include burning System Configuration eFuses, such as one to disable JTAG access. It is also critical to disable the U-Boot and UART console to limit access to the device consoles.

Furthermore, for a system to be truly secure, it must be capable of adapting to new vulnerabilities, which can be accomplished with a secure software upgrade mechanism. Security is paramount in software upgrade mechanisms to ensure bad actors cannot exploit the system to their advantage.

5:41 – Importance of Software Upgrades

All of this is especially true in embedded systems that implement mission-critical applications, making them highly sought-after targets.

Software upgrade is a key component in secure embedded systems. Without the ability to upgrade a system, there’s no way to fix vulnerabilities after a device has been deployed. Software upgrade is a sensitive process, and it can expose vulnerabilities since it interacts directly with the hardware responsible for firmware and software storage. For example, if a problem occurs while writing the update to flash memory, such as a power failure, then the flash memory write may fail, resulting in the device being unbootable.

In addition to the sensitive nature of programming device firmware, software upgrades also expose another attack vector that can be exploited by bad actors. An embedded system implementing an unsecured software upgrade mechanism could allow attackers to utilize the mechanism to reintroduce system vulnerabilities that were intended to be blocked by securing the system. These vulnerabilities are mitigated by employing security best practices in the software upgrade mechanisms to help minimize the attack vectors that software upgrades expose.

7:22 – High-Level Characteristics of Secure Software Upgrades

Let’s go through some of the high-level characteristics of systems that implement a secure software upgrade mechanism. Boot process security is the foundation for platform security. In a previous webinar, we covered the mechanisms used by Secure Boot to establish the system’s Root of Trust by authenticating and encrypting bootloaders. Once Secure Boot has established the Root of Trust, it can be extended to components later in the boot process, again through authentication and encryption.

Later, we will touch on how the Root of Trust can be used during secure software upgrade mechanisms by including it inside the Root of Trust.

7:32 – Trusting Firmware During Software Upgrades

Trusting the firmware or software executed on the device is another key component of systems that allow for software upgrades. Boot firmware can be validated via measurements. These measurements can then be validated by the system to ensure that the software executed is as expected. Measuring software components is accomplished by hashing the memory containing the application that is about to be executed. Hashes are one-way functions that create unique, fixed-length values based on variable input.

The hashes generated act as a unique fingerprint or measurement of the software that is about to be executed. These measurements can then be sent to a remote attestation server, where they will be compared with expected measurements. The server will then indicate whether the measurements are valid or not. Attestation of the measurements can also be done locally with signed good measurements located on the device or using an external chip called the Trusted Platform Module (TPM).

8:30 – TPM and Platform Configuration Registers (PCRs)

At a fundamental level, Trusted Platform Modules (TPMs) can keep track of platform measurements using internal PCRs (Platform Configuration Registers). While keeping track of these measurements, policies can be created to store secrets against the values of these PCRs.

This allows keys to be locked when the system is in an untrusted state, such as in Linux user space or if the boot firmware has been modified, but be unlocked while the system is in an authorized state, such as in the early bootloaders. Common secrets stored in the TPM are RSA private keys and AES encryption keys, and these are released only when the system is running specific software with specific PCR measurements.

9:09 – TPM as a Secure Key Storage Mechanism

Furthermore, the TPM can be used to encrypt or decrypt data without exposing the AES keys, in the same manner as an HSM (Hardware Security Module). For example, during device manufacturing, the TPM can be programmed with a secret AES key used for decryption during the boot process. Then, when the TPM PCRs are in a specific state, the TPM will be allowed to use that pre-programmed key to decrypt the data without ever exposing the secret AES key.

9:36 – Sealing Secrets in the TPM

Sealing secrets in the TPM against PCR values is only one method for storing keys provided by the TPM. Another common approach, especially in software upgrades, is to use what’s called an Authorized Policy to seal secrets. With authorized policies, secrets are stored in the TPM with an accompanying public key. Access policies can then be generated and signed with the accompanying private key to define access conditions for unsealing the secret.

10:00 – Authorized Policy in Software Upgrades

This mechanism is extremely important for software upgrades, where PCR values change after a software upgrade. In this case, when the build server has generated the new firmware and software components, the new PCR values can be calculated for those components. The build server can then create a policy with the new values and sign it with the private RSA key. During the update process, the device can substitute the new policy for the old one. Then, during the next boot, the new firmware will be booted, resulting in new PCR values. Since there is now a new policy that reflects those PCR values, the board will boot as usual.

In turn, this process separates the TPM secret sealing process from the access policy creation, which is very convenient for software upgrades where PCR measurements change frequently.

10:58 – Securing the Software Upgrade Process

Another aspect of securing the software upgrade process is being able to rely on the security implemented by the software upgrade application. Developing a robust application depends on the location of its implementation. One option is to implement the upgrade application in user space with very limited permissions. Although this solution leads to a simpler development and upgrade process, the upgrade application is vulnerable to operating system exploits.

The Trusted Computing Group guidance for secure upgrade of software and firmware recommends implementing the software update mechanism inside of a firmware upgrade latch. The firmware upgrade latch is a mechanism that prevents the software from being modified outside of the latch.

12:38 – Firmware Upgrade Latch in Secure Systems

One approach to implementing a firmware upgrade latch is to include the firmware upgrade application inside the Root of Trust as part of the boot process. This comes with the caveat that bootloaders are not always updated as part of the software upgrade mechanism, which means that a separate update application may be required to update the actual upgrade application. But this also means that the software upgrade application can be trusted since it’s part of the Root of Trust.

Software upgrade applications are a common attack vector used by bad actors to exploit systems since they have direct access to modify the firmware or software running on the device. One way that attackers may try to exploit the software upgrade application is through the upgrade package itself. For example, if the upgrade application does not provide authentication or integrity checks on the upgrade payload, attackers may be able to load their own software on the device.

Even with authentication and integrity checks on the upgrade payload, attackers may also opt to leverage valid payloads to create vulnerabilities in the system. This is typically accomplished by attempting to upgrade the system with an older version of the software to reintroduce vulnerabilities that were previously patched.

13:50 – Downgrade Protection in Software Upgrades

To solve this, software upgrades must ensure that the software being updated is a newer version than the one running on the device. Although encryption of the upgrade payload may not be required, it can be useful to implement to add one extra layer of security to the system. Since eliminating all vulnerabilities in a system is nearly impossible, encrypting the upgrade payload makes the upgrade mechanism more difficult to reverse-engineer.

In security, this strategy is called layered security, where encryption may not be necessary to protect the data, but it is useful to further obfuscate the process from bad actors.

14:47 – Disabling Insecure Recovery Methods

In a secure system, many of the mechanisms that could have been used for system recovery are inaccessible. As was explained at the start of the webinar, the U-Boot console should be disabled, meaning it cannot be used to re-flash the firmware to the device. In the case of a software upgrade failure, many other common methods should also not be accessible for security reasons, like JTAG.

It is for this reason that some sort of recovery system is necessary to recover the system when the software upgrade mechanism fails, either from a bad payload or interruption due to power loss.

15:22 – Common Recovery Mechanisms

Hardware watchdogs are used extensively for system recovery by catching failures in the boot process and causing the system to re-enter an unsuccessful boot after a software upgrade. When a boot limit is reached, a recovery mechanism is usually initiated to recover the system from a non-functional state.

Common strategies for system recovery include firmware recovery images and A/B upgrade mechanisms. System recovery can be accomplished via a minimal firmware recovery image that boots after a failed software upgrade. This minimal image should allow for the delivery of an upgrade package and initiate the upgrade mechanism with upgrade applications located in the boot firmware.

15:51 – A/B Upgrade Mechanisms

Another strategy for system recovery is an A/B upgrade mechanism. A/B upgrade mechanisms store two full versions of the software on the device. The main flow works as follows: let’s say the system is booting version A of the software and is prompted for a software upgrade. The device will take the upgrade payload and program it to version B of the software. Boot parameters will then be updated to indicate that version B should be booted, and the system is in a software upgrade state.

If version B of the software is booted and reaches a boot limit, which is a boot counter that increments every time the device is started, the software upgrade will be indicated as a failure, and the device will seamlessly switch back to version A of the software. But if version B boots correctly, it indicates through the boot parameters that the software upgrade is complete, and version B becomes the official version of the system.

16:17 – Advantages and Trade-Offs of A/B Upgrade Mechanisms

An advantage of A/B upgrade mechanisms is seamless system recovery by switching back to the previous version of the system. However, A/B upgrades require the device to be capable of storing two versions of the software, essentially meaning that double the amount of storage space is required.

Boot reversion logic is another vulnerable aspect of the system since bad actors could revert to the unused version of the system to reintroduce vulnerabilities. There is a trade-off between development time and the location of the A/B version split. Typically, the A/B version split happens in the second-stage bootloader, such as U-Boot, due to the available drivers.

The downside of splitting in U-Boot is that all components before and including U-Boot are not part of the A/B upgrade system. This doesn’t mean they cannot be upgraded; it just means that if upgrading these components fails, the system cannot be recovered. In some cases, it is possible to move the A/B version split before the first-stage bootloader through creative use of multi-boot software sets and bare-metal applications. However, the complexity of the system requires much more development effort.

17:20 – Software Update Frameworks

Frameworks exist for out-of-the-box software upgrades, such as Mender, RAUC, and FWUP. These frameworks offer support for A/B upgrades and over-the-air (OTA) updates. For example, Mender provides U-Boot drivers for A/B upgrade versioning as well as OTA upgrades. These frameworks are not all open-source, so they may require licensing for commercial applications. However, they can be circumvented with custom solutions tailored to the specific requirements of the system, including TPM support or early A/B version splitting.

17:58 – Encrypted File Systems in Secure Systems

Encrypted file systems are a key component of secure systems, protecting applications and customer data. Dcrypt LUKS partitions are a common solution for implementing this. Dcrypt sits between the read and write operations to perform encryption and decryption on the fly for the data held inside LUKS partitions.

In addition to file system encryption, it is common to pair this functionality with a TPM to hold the secret used to unlock the file system. This ensures that the device firmware and software, measured during boot, are correct prior to releasing the key and allowing the file system to be mounted.

If the secret used to decrypt and mount the file system is shared between the build server and the device prior to a software upgrade, this mechanism comes with the advantage that the software upgrade doesn’t need to know the secret. Instead, the encrypted file system can be passed in its entirety to the device and used to overwrite the old partition.

The downside of this approach is that the upgrade package becomes quite large, as the encrypted partition cannot be compressed well due to high entropy.

18:59 – Demonstration of TPM in Action

Now, let’s go over an example of storing a secret in the TPM to be used by the system when it is in a trusted state. We will go over storing the secret in the TPM and then demonstrate how altering PCR values makes the secret inaccessible. Currently, on screen is a UART console connected to a KR260, an MPSoC platform that includes a TPM.

The FSBL (First Stage Boot Loader) has been patched to extend TPM PCR values with early A/B component measurements. Using the TPM2 Tools package in user space, I can interact with the TPM using the PCR Read command. We can see the current state of the PCR values:

• PCR0 is extended with a measurement of the boot ROM,
• PCR1 is extended with a measurement of the first-stage bootloader,
• PCR2 is a general PCR that’s extended with many measurements, including the system device tree,
• PCR3 is extended with a measurement of U-Boot,
• PCR6 is extended with a measurement of the system bitstream.

19:48 – Sealing a Secret in the TPM

Next, let’s go over sealing a secret in the TPM. In this case, we’ll be storing the secret contained in secret.txt, which is “hello software upgrade webinar.” I’ve created scripts to help me during this webinar—one that creates policies, one that seals secrets, and one that unseals secrets. So, let’s go ahead and seal the secret in the TPM.

You can see that it’s sealing the secret, and it gives the name and value of the secret being sealed, which could, in reality, be an RSA or AES key. In this case, it’s just text. What the TPM is doing is sealing the secret using an Authorized Policy. Before this webinar, I generated an RSA key pair and provided the public key component and the secret to the TPM for storage.

20:17 – Sealing a Secret in the TPM

Dawson Theroux:
I’ve created scripts to help me during this webinar: one that creates policies, one that seals secrets, and one that unseals secrets. So let’s go ahead and seal the secret in the TPM.

As you can see, it says it’s sealing the secret, and it gives the name and value of the secret being sealed, which could, in reality, be an RSA or AES key, but in this case, it’s just text. And what it’s doing is sealing the secret using an Authorized Policy. Before this webinar, I generated an RSA key pair and provided the public key component and the secret to the TPM for storage.

Next, we can create a policy to unseal the secret.

The policy that’s being generated is a PCR Authorized Policy, which means that it takes the current state of the PCR values and creates a policy. Then, the policy is signed using the private key component of the RSA key pair.

21:30 – Unsealing the Secret

Now, we can unseal the secret using this policy. Let’s go ahead and unseal the secret. You can see that the script prints the correct value. So, what the TPM is doing in this case is taking in the new policy that we just created, checking the signature, and ensuring it matches the RSA public key that the secret is stored with. If that authentication passes, it will then check the policy value, which is the current state of PCRs 0, 1, 2, 3, and 6. If all of that passes, it will return the secret.

22:35 – Modifying PCR Values

Let’s simulate what happens if someone modifies a component of the system. We can do that by extending a PCR with a random value. In reality, this could mean someone changes the boot.bin file, which results in different values being measured, but for this demonstration, we will simply modify PCR 0.

I’ll extend PCR 0 with a new value—just a string of zeros in this case. If we read back the PCR values now, we can see that PCR 0 has been modified. It now starts with a different value than it originally had.

Now, if we try to unseal the secret with our original policy, it no longer works.

As you can see, the policy check failed. This is because, although the signature of the policy is still valid, the PCR values that the policy was originally created with no longer match the current state of the PCRs. This means that if someone tampers with the system, the secret can no longer be unsealed.

23:30 – PCR Changes During Software Upgrades

In some cases, like during a software upgrade, it’s expected that PCR values will change. For example, maybe you’ve upgraded the first-stage bootloader, or any other early boot component, and the PCRs have changed as a result. Luckily, policies can be created on the build server, which knows what the new PCR measurements will be after the upgrade.

The build server can predict what the PCR values will be after the upgrade and generate a new policy to match those new values. It can then send that policy to the TPM to be used after the upgrade, so when the PCR values change, the new policy will still allow the TPM to unseal the secret.

24:10 – Reverting PCR Values

Since hashes are a one-way operation and are irreversible, it’s computationally infeasible to reverse the hash and go back to the original value of the boot ROM measurement. The only way to revert the PCR values to their original state is to reset the TPM by toggling the reset line. On most systems, like the one we’re working with here, the reset line is connected to the Power On Reset button on the device.

Once the system is reset, the first-stage bootloader will re-extend the correct values to the PCRs, and the TPM will go back to its original state.

After the reset, you can see that PCR 0 has returned to its original value, and now the original policy can be used to unseal the secret once again.

25:00 – Demonstrating TPM Usefulness

I hope this demonstrates the usefulness of a TPM in a secure software upgrade system. The TPM can store secrets and only release them when the system is in a trusted state. This ensures that even if someone tampers with the system, those secrets remain protected.

26:02 – Conclusion and Q&A

Alecia:
Thank you, Dawson, for that great presentation and demonstration on how to implement secure software upgrades in embedded systems. We will now move on to the Q&A portion of today’s session. If you haven’t already, feel free to type in your questions, and we’ll address them one by one.

Our first question comes from John. John asks:
“Can the TPM be used in conjunction with a software-only Root of Trust, or is it hardware-specific?”

Dawson Theroux:
That’s a great question. The TPM is typically hardware-specific, as it is a hardware module that provides cryptographic capabilities and secure storage. However, you can implement a software-based Root of Trust, but it’s generally considered less secure because software is more prone to tampering than hardware-based solutions. The benefit of using a TPM is that it provides a hardware Root of Trust that is much more difficult to compromise. So while software-only Roots of Trust are possible, they aren’t as robust as hardware-backed options like TPMs.

27:15 – Second Question

Alecia:
Great, thanks Dawson. The next question is from Sarah. Sarah asks:
“How do you handle recovery if both the A and B software versions fail in an A/B upgrade mechanism?”

Dawson Theroux:
That’s a good one. If both A and B software versions fail, you typically need some sort of minimal recovery image that can be used to restore the system to a functional state. This minimal recovery image should have enough capability to receive a new upgrade package and initiate the upgrade mechanism. The minimal image is usually stored in the boot firmware or as part of the device’s ROM so that it’s always available, even if the A/B software versions both fail. This ensures that there is always a fallback option to recover the system.

28:05 – Third Question

Alecia:
Next, we have a question from Robert. Robert is asking:
“How does TPM handle rollback protection during software upgrades?”

Dawson Theroux:
Great question, Robert. Rollback protection is one of the main benefits of using TPM during software upgrades. The TPM measures key components of the system, such as the bootloader and firmware, and stores these measurements in PCRs. If someone tries to load an older version of the software, the PCR values won’t match, and the TPM will refuse to release any secrets or keys required to boot the system. This effectively prevents the system from booting older, vulnerable versions of the software, providing strong rollback protection.

28:55 – Fourth Question

Alecia:
Our next question comes from Lisa. Lisa asks:
“Is it possible to perform over-the-air (OTA) updates securely using TPM?”

Dawson Theroux:
Yes, absolutely. OTA updates can be secured using a TPM. The TPM can be used to verify the integrity and authenticity of the update package before it’s applied to the system. It can also ensure that the upgrade process follows the proper chain of trust. By using TPM, you can protect the update process and ensure that only verified updates are installed on the system. Many frameworks, like Mender and RAUC, support TPM integration for securing OTA updates.

29:40 – Closing Remarks

Alecia:
Okay, well, we’ve actually addressed all the questions. Unless anyone has additional ones, I’d like to give a quick shout-out that Fidus will be attending the first-ever Embedded World in North America, happening this October in Texas. If anyone on this webinar is planning to attend, please stop by and say hello! We’ll be located near the Lattice booth, one of our key partners.

Also, as Dawson mentioned earlier, we have another webinar on a topic related to secure boot, available on our website. We’ll be sending out the recording of today’s webinar by email tomorrow once it’s uploaded, and I’ll also include a link to that previous webinar for those interested.

Finally, if you have any suggestions or topics, you’d like us to cover in future webinars, please don’t hesitate to reach out and share your feedback.

Dawson, thanks again for your time and for putting this together. It’s always great to have you with us. And thank you to everyone for joining today. We’ll talk to you soon. Bye!