Z-Wave Firmware Hardening

This is an abbreviated version of the full blog on the Z-Wave Alliance website.

Every IoT device has bugs. Today’s devices have many thousands of lines of code and too many hardware features meaning there are plenty of bugs hiding in every device. Bugs are in your product. I know it, and you know it. I propose that the best solution for these bugs is to “harden” your firmware to make it more resilient and keep on truckin’ if something bad happens. Below are my Tips and Techniques for hardening Z-Wave firmware to survive a failure. These ideas are for Silicon Labs SDK but similar techniques apply to Trident IoT.

Seven Tips to Harden Z-Wave Firmware

1. Assume Everything is Broken
2. Use FOR Instead of WHILE
3. Replace Default_Handler
4. Enable the Other Watchdog
5. Reboot if no Comms in a Day
6. Enable Stack Overflow Checking in FreeRTOS
7. Run Static Analysis Tools

1. Assume Everything is Broken

This is a philosophical idea you need to keep in the back of your mind with every line of code you write. Assume everything is broken all the time – hardware never goes “ready”, a queue is always full, a mutex never switches, an impossible state occurs and similar sorts of failures. The most common code technique is to always check for error conditions of any function that returns a value. Always check inputs for validity.

The most insidious failures are stack overflows. This can happen where parts of RAM are overwritten and I’ve found it amazing that the code can keep running even after trashing potentially hundreds of memory locations. The challenge is there is no way to predict what might happen. All sorts of things that “can never happen” absolutely will happen when the stack overflows. Due to the limited RAM on Z-Wave chips, it is easy to overflow the stack.

Another common impossible condition is when the power supply sags just enough to flip bits in ways that are technically impossible. Strong magnetic fields from nearby motors and even cosmic radiation can flip bits in impossible ways. There is truly nothing that “can’t happen”. Thus, always code with the thought that the impossible can happen because eventually it will.

Wireless IoT devices using Z-Wave are often wired directly to mains-power. They cannot be easily rebooted like you do with your computer when it freezes. If a device bricks, it’s dead potentially for months or even years before a power failure brings it back online. Once a device is wired in, it’s usually there for many years and with Z-Wave it can be decades. Ensuring the firmware is resilient when (not if!) the impossible occurs will keep customers happy since they never knew the device rebooted – it kept on truckin’.

2. Use FOR instead of WHILE

The Silicon Labs SDK, including the bootloader, has many while(hardware_busy) loops that will wait forever and can cause the device to brick. For example, in Silicon Labs if you enable the LFXO (32KHz crystal oscillator) but don’t have the crystal wired up, the startup code waits for the LFXO to be “ready” with a while loop. While this is obvious when debugging firmware, this loop causes a device in the field to brick if for some reason the crystal stops working. The simple solution to this is to add a FOR loop with a timeout enabling the code to continue.

Example in em_cmu.c:

Replace: while ((LFXO->STATUS & _LFXO_STATUS_ENS_MASK) != 0U) {  }

With: for (int i=0; (i<1000)&&((LFXO->STATUS & _LFXO_STATUS_ENS_MASK) != 0U); i++) {__NOP()}

Note the __NOP() is necessary to prevent the compiler from optimizing the loop and removing it. The timeout value (1000 in this case) must be chosen based on testing. I usually set it to 10X the typical value. Following the FOR should be an assert to check that the timeout didn’t occur. Note the use of < and not == for the check of the timeout. If the impossible were to happen, there is a chance “i” could skip past exactly 1000 and then since this is a 32-bit number, the timeout would be waiting for a long time for the 32-bit number to wrap all the way around. This is another defensive coding technique where in the back of my mind I’m thinking of the impossible and coding to be resilient even when the impossible happens.

3. Default_Handler

Segger has a great article on debugging the many “fault handlers” in the Cortex-M processors. The article provides code for many different handlers to help debug the fault and make the code more resilient.

Default_Handler is in startup_<chipnumber>.c and is unfortunately NOT declared as weak so you must edit the SDK file itself. Best to select the Copy Contents mode in the .slcp file Import Mode. Maybe it’s better to fix in the SDK for all your projects! The Silicon Labs SDK has only a single line while (true); for a default handler. This code counts on the watchdog to eventually reboot the chip. But the reboot isn’t guaranteed and there is no additional debugging information as shown in the Segger examples. All the fault handlers are mapped into this single handler, but they can be individually overridden due to weak assignments. Even something as simple as a divide by zero can cause a fault handler to be called and brick the device.

At a minimum, put the Segger recommended code in for at least some of the exception handlers to make debug easier. Generally, it is a good idea to light an LED (ideally red) or some other external indicator to help during debug. Other ideas are to log the address and condition that caused the exception and store it in the User Data Page/NVM that can be read out from production units that were returned from the field by angry customers. Then perform forensic analysis to identify the cause and release a firmware update that solves the problem.

4. Watchdogs

Watchdog timers are crucial for reliable 24x7x365 operation of an IoT device. A watchdog timer is a timer that slowly counts down. Every now and then, the firmware “feeds” the watchdog by resetting the counter to a high value. If the counter reaches zero, a full reset of the chip is triggered which reboots the chip and hopefully resolves the error condition. The watchdog timer typically takes a couple of seconds of being starved before the reset to ensure it doesn’t falsely reset. The trick to a resilient watchdog is deciding when to feed it, and more importantly, when not to. I wrote a blog post on watchdog timer best practices back in the 500 series days which still applies.

5. Reboot When no Communication for a Day

Hello, is anyone listening? The concept here is basically a long-duration watchdog timer. If the controller hasn’t sent a frame and/or hasn’t acknowledged the receipt of a frame in twenty-four hours, maybe a reboot will clear things up. I’ve seen this in the 500 series where on rare occasions reading the HomeID from the external NVM would fail. As a result, the device would forget the HomeID and assume some random number. This random number would then be stuck in the device for days or weeks or even years until the device rebooted for some reason. This was a classic Impossible Condition that seemed to happen on a fairly regular basis when several tens of thousands of Z-Wave devices have been operating for a few months. The only solution for the end-customer was to rip the bricked device out of the wall! Or more commonly factory reset it and rejoin the network which resulted in one-star reviews.

The solution is simple, setup a 24-hour software timer and check the RX/TX statistics and if nothing has made it through, reboot! Rare impossible conditions are fixed in a way that customers never notice. This check is only needed for always-on or FLiRs (LSEN) devices as deep sleeping devices reboot every time they wake up.

6. Stack Overflow Checking

A real-time-operating-system adds complexity, but FreeRTOS has a feature to check for a stack overflow which is enabled by default. The variable configCHECK_FOR_STACK_OVERFLOW is set to 2 by default in FreeRTOSConfig.h. This enables some checking and fills the stack space with 0xA5s which is then checked with each task switch. Inspecting RAM after running the code for some time in the debugger can provide insights as to how close to overflowing the stack has happened so far. The check calls vApplicationStackOverflowHook if there is a failure but there is only an assert in the weak function. My recommendation is to add a breakpoint here during testing and consider rebooting in the released code. Stack overflow checking is only recommended during development and testing due to the additional overhead.

The insidious problem with stack overflows is they often require several things to go wrong at the same time – a task switch, an interrupt, the radio sending or receiving data and having to find a new mesh route, code allocating sizable temporary buffers and maybe even more code that uses up the limited stack space. As mentioned above, I have observed the stack overflowing, trashing many dozens of memory locations and the code keeps running but eventually there is an impossible condition or more often a hardfault exception. As a result, the failure is often overlooked as it only happened that “one time” but in reality, it happens a lot. Getting the failure to happen repeatedly in a controlled environment is often very difficult. I have had dozens of units set up testing a specific failure case which would take all weekend to finally trigger. Then not having enough data on the unit that failed makes it even more exasperating.

7. Static Code Analysis

Use Claude, CodeX or other static code analysis tools to review all firmware. AI continues to improve at an exponential rate to grade code quality. Often AI can recommend changes to fix the code, but I would carefully check over the suggestions. AI can hallucinate or simply start making things up out of nowhere. The GCC compiler has a -fanalyzer option that will find a few interesting things. Coverity is an industry leader in this field but is pricey. What tools have you used?

Next Steps

Part 6 of the Z-Wave Developer’s Journey discusses hardware best practices. I present my tips and tricks for making low-cost, easy to debug and manufacture Z-Wave products from my 25 plus years of Z-Wave experience. As we continue along the Z-Wave Developer’s Journey, I welcome your comments and questions.  Please feel free to reach out to me directly via email.