A Z-Wave Developer’s Journey | Part 7

DrZWaveLeave a comment

Optimizing Z-Wave IoT Devices to Achieve 10+ Year Battery Life

Z-Wave sensors can run for 10 years on a CR2032 coin cell

This is an abbreviated version of the full blog on the Z-Wave Alliance website.

Introduction

Wireless IoT devices are typically powered with batteries. Thus, a key design criterion is: how long will the batteries last? Z-Wave developers must balance battery lifetime, battery type and functionality. Simple Z-Wave sensors can run on a single coin cell for as many as ten years. Door locks need four AA batteries to mechanically throw the locking bolt, but the electronics contribute significantly to battery life. Most devices need at least a two-year battery life otherwise customers feel they are constantly changing batteries resulting in a one-star review.

There are three components of battery life calculation:

  1. Average current while sleeping
  2. Average current while awake
  3. Awake duration

Battery lifetime calculation: Blifetime = Bcapacity/((Iawake * Tawake * Nwakeups) + Isleep)

For example: A device with 3 AAA batteries has a capacity of 1000mAhr, an average awake current (Iawake) of 12mA for 0.2 seconds (Tawake) with 6 wakeups per hour (Nwakeups) and an average sleep current (Isleep) of 0.030mA, you get 1000/(12*(0.2/3600)*6+0.03=29,000 hours or 3.3 years. Thus, the key for long battery life is to minimize the average sleep current and limit the number of times the device wakes up. Obviously matching the size of battery to the task at hand is crucial. Too small of a battery and the user is annoyed with replacing it too often, too large of a battery increases the physical size of the product and the cost.

Troubleshooting

The classic problem when debugging a battery powered device is isolating a high current draw. When debugging a new design, the typical problem is that the board is drawing hundreds of microamps (or even many milliamps!) when it should be under a few tens of microamps. The obvious first thing to check is if the CPU is sleeping. Load the firmware into a devkit so it is the only device drawing power. This isolates all the other sensors or circuits on the board so at least you know the CPU is asleep. Here’s a brief checklist for high current debug:

  1. Is the CPU in Sleep mode?
  2. Is the high-speed oscillator off?
  3. Are GPIOs floating? Unused inputs might oscillate which can draw a lot of power
  4. Are GPIOs fighting pullups/pulldowns? Try disabling pullups/pulldowns in the debugger
  5. Are other circuits drawing power?
    1. Cut power traces, lift power pins, lift circuit pins, remove circuits from the board
  6. Debug GPIOs causing phantom power draw?
    1. Disconnect ALL debug connections – SWD/JTAG and even ground

The big problem comes when customers start returning the product because of short battery life. You may have optimized the battery life in the lab, but did you test it in the real world? One thermostat I worked on would constantly send a humidity report because the humidity reading would bounce back and forth by one percent with each reading. Never saw this in the lab but it became quite common in the field. This was easily solved by adding hysteresis to the readings and requiring the humidity to change in one direction by more than three percent before sending a report. Fortunately, we were able to OTA the firmware and solve the problem before it became widespread.

A serious problem with early FLiRS (now called LSEN) door locks didn’t become a problem until many locks on many Z-Wave networks were installed in a single building. The original WakeUp beam only had the NodeID of the door lock to wake up. The problem was that with many Z-Wave networks in close proximity, every door lock having NodeID 02 (which was the case on all of the networks in the building) when the controller woke up their door lock, all the door locks on all the adjacent networks would wake up because they were all NodeID 02. Battery life in many cases was only a few months due to the gateway polling each door lock every night which meant every door lock woke up hundreds of times every night. This was solved years ago by adding a hash of the HomeID to the WakeUp beam which pretty much assures only the desired node will wake up.

A particularly difficult battery life problem happened with a door/window sensor that in rare cases would remain awake. We had to set up a dozen units and watch the power on them all individually and when one stayed awake for more than 60 seconds then we captured data and stopped the test so we could debug the problem. Turned out the stack was overflowing, causing impossible conditions as I mentioned in Part 5 of the Developers Journey. Often the device continued to function, but it just didn’t sleep causing the battery to die in a few days. Turned out the cause was a complex interaction of interrupts and the Z-Wave mesh rerouting at the same time which is why we didn’t see it in lab testing. Masking all interrupts while servicing the sensor and some careful adjustments to the stack sizes solved the problem which was OTAed to all units in the field.

Remember that each radio transmission uses a significant amount of power. Keep all communication to a minimum especially when powered with coin cells. Ensure that all possible values of configuration parameters can’t accidentally cause rapid transmissions.

Tools For Measuring Power Consumption

Every Z-Wave firmware developer needs access to power measurement tools. These tools can range from Silicon Labs AEM which is part of the development kit to expensive laboratory tools. Measuring the current draw versus time and a few GPIOs is critical for rapidly optimizing battery life. Bill Hewlett, co-founder of Hewlett Packard, has a popular quote: “You cannot manage what you cannot measure… and what gets measured gets done.” The list below provides some tools I’ve used for optimizing battery life – let me know what you’ve had success with!

5-1/2 digit Digital Multi-Meter (DMM) or Bench Power Supply

DMMs with at least 5-1/2 digits have enough dynamic range to measure milliamps to nanoamps. Modern DMMs come with USB and can be remotely controlled and record waveforms for voltage and current but typically lack additional channels for GPIOs or UARTs. DMMs are super accurate for very low currents which is why I typically use one to precisely measure deep sleep current. A good DMM costs $250+ but there are multi-channel power supplies that have similar capabilities for about the same cost. I often use a cheap DRoK power supply with an OLED display to know when the device is sleeping or not, when I don’t need nanoAmp accuracy.

TinyCurrent

TinyCurrent is a budget tool for measuring current but has some limitations.

The TinyCurrent is a cute and cheap device to convert current into voltage which you then connect to your scope. This is a simple opamp across three switch selectable shunt resistors to select the range. The switch is the main limitation of this device as the dynamic range isn’t enough to cover the milliamp awake mode to the sub microamp sleep mode transition. Since it’s a simple analog conversion of current to voltage, it is easy to connect to a scope along with several GPIOs. Measuring the deep sleep current must be done with a DMM or other device but this works fine to quickly see when the device is in deep sleep or active modes.

Nordic PPK

Power Profiler 2 is a capable current measurement tool

The Nordic Power Profiler Kit II (PPK2) is a cost efficient and capable power measurement tool. The nRF Connect windows app displays the current graphically along with up to 8 digital GPIOs. PPK2 measures between 200nA and 1A at 100ksps. The PPK can supply up to 1A of 3.3VDC but I found the supply isn’t clean enough to measure small currents accurately. Instead, I use a battery (usually 3 AAA in series) and pass that through the PPK to get a clean power supply. As a side benefit of using real batteries, the voltage sags during high current draw since the power source is an actual battery. The high dynamic range is achieved by switching between 3 shunt resistors which take several samples to switch. Unfortunately, this is often exactly the behavior that is of interest. Overall, this is a capable tool for an affordable price.

Silicon Labs AEM

Silicon Labs Advanced Energy Monitor (AEM) is the primary power measurement tool for Z-Wave developers. A WSTK (BRD4002) developer’s kit is required to run the AEM which you can now purchase by itself instead of part of a more expensive kit. The AEM supplies 1.8-3.6v up to 500mA with a 100kHz sample rate and a resolution just under 1uA. There are four digital logic inputs as well as the VCOM UART. The Energy Profiler is a tool within Simplicity Studio that provides a graphical interface for the AEM. The energy profiler tool suffers from stability problems but is capable if you are working with Silicon Labs Z-Wave chips.

JouleScope

Joulescope JS220 measuring the current of Z-Wave devices during transmission and while sleeping

Joulescope is currently the industry standard for IoT development. While pricey, it is the perfect tool for the job. Full disclosure: Joulescope provided me with a JS220 for a previous project which I now use for every project. Wide voltage range of +/- 15V, 3A to 0.5nA resolution, 300kHz bandwidth and a reliable multiplatform software application make the Joulescope indispensable. The JS220 has 4 digital inputs and 2 outputs which help correlate current draw with the firmware being executed. The newer JS320 version is now available with even more dynamic range. The screenshot shows the dynamic range from microamps to over 400 milliamps and two digital logic signals indicating the state of the CPU.

Oscilloscope Current Probe

Traditionally, an oscilloscope current probe was used for measuring current and simultaneously sampling several analog or digital channels. The problem with most of these probes is they were designed for high current (60A or more) and relatively low bandwidth and are very expensive, costing around a thousand dollars just for the probe without the scope. They do not have the dynamic range to go from nanoamps to milliamps and generally don’t have the resolution below milliamps. Maybe there’s one in a drawer somewhere you can use but I wouldn’t buy one of these for debugging IoT devices.

Next Steps

Part 8 of the Z-Wave Developer’s Journey discusses sub-GHz antennas which can make or break a Z-Wave product. The distance a Z-Wave product can be from the hub is determined by the efficiency of the antenna. I’ll discuss a few key factors in low-cost antenna design from my experience of having deployed many different types of antennas. As we continue along the Z-Wave Developer’s Journey, I welcome your comments and questions.  Please feel free to reach out to me directly via email.

A Z-Wave Developer’s Journey | Part 6

DrZWaveLeave a comment

Z-Wave Hardware Best Practices

This is an abbreviated version of the full blog on the Z-Wave Alliance website.

Scattered debug pins require a custom jig which is fragile and easily lost. Standard debug connectors are reliable and easy to replace.
Use a standard debug connector for reliable maintainable IoT Z-Wave designs

Introduction

Cost sensitive wireless smart home devices have different requirements compared to designing cell phones or laptops. The printed circuit boards (PCB) are usually only a few square inches and often only two layers. To fit inside of an electrical outlet box the entire product must be small and cost considerations require integrated microcontrollers to supply most of marketings requirements.

Basics of IoT Device Hardware

A key feature of having the engineer do the PCB design, is that the engineer can easily change the MCU pinout to solve PCB layout problems. Since both Silicon Labs and Trident IoTs Z-Wave chips can route nearly any peripheral to almost any GPIO, I often swap GPIOs while routing the PCB. Make a quick change to the schematic, reread the netlist in the layout tool and voilà, problem solved! Just remember to also change the pinout in the firmware.

I usually manually design, layout and hand-assemble prototypes. If I need to build more than five, I contract assembly out because assembly quickly becomes a huge time sink. I use the open source KiCAD electronic design software. I have contributed a few library components to the repo including the Silicon Labs ZG23 and Tridents CZ20. Many PCB vendors can directly import KiCAD files, but I prefer to send them good ol’ Gerber files to ensure I get exactly what I want. With the KiCAD files, the vendor may adjust parameters such as the solder mask clearance which results in no solder mask between the fine pitch pins of the MCU resulting in nothing but solder shorts between the tiny pins. I don’t even use the autorouter as it tends to make a mess of things with way too many vias. The routing is usually simple enough to complete in a few hours and is an enjoyable game of close-the-rats-nets.

Keeping Costs Low

The main cost driver for consumer electronics is volume. The higher the annual volume, the lower the cost for nearly every line item; PCB manufacturing, the MCU, passives, ICs, assembly, test and packaging. If the product volume isn’t at least in the 10K units/year range, then it must be unique enough to command a high price. There is a significant knee-of-the-curve at the 10k threshold.

Don’t spend too much time checking, rechecking, and reviewing the schematic and layout. The turnaround time for a new prototype PCB can be as little as overnight, and the cost of a new board is just a few dollars. Thus, it makes sense to send the board out quickly and expect to spin it once or twice to fix a couple of errors. I accidentally put the wrong size passive component on the board and only discovered the mistake when it barely fits in the footprint on the board. But that’s easily fixed and a new board is ready next week. I’ve also reversed the pinout of a connector or some other special component that gives the bottom view in the datasheet instead of the top view. Again, easily fixed and I have new boards in about a week. Be careful with GPIO selection as the 800 series has limitations on power domain and wakeup capabilities of certain pins. See my GPIO blog for more details.

Rules for low-cost IoT products

Naturally these “rules” are not absolute requirements but instead are recommendations. Engineering is the art of balancing the solution to meet the requirements in a cost-efficient manner. Often it is worthwhile to push back on a requirement if the cost is too high so don’t simply accept the requirements as non-negotiable. The first rule here is an example of these engineering tradeoffs: a four-layer PCB is more expensive, but you get better impedance (thus better RF) and likely a smaller PCB. The balance between 2 vs. 4 layers is unique for every product which means it must be engineered.

Rectangular PCBs are lower cost for your wireless Z-Wave IoT product. Use two layers instead of 4 and keep most components on one side.
Two layer rectangular PCBs are cheaper
  1. Minimize PCB layers
    1. 2 works for most Z-Wave designs
    2. 4 costs ~40% more for bare PCB but has better impedance/RF
  2. Minimize size
    1. PCB costs are computed by the square millimeter
    2. Remember the GND plane of the PCB is half of the antenna
  3. Rectangular outline
    1. Simple shapes maximizes array utilization
    2. Ideally v-scored for minimal waste
  4. 0.16-0.20mm (6-8mil) wire spacing
    1. Follow the design rules of the low-cost PCBs manufacturer
  5. Minimize special holes
    1. Slots, unusual sizes, edge features all cost $
  6. All SMT components on one side – cheaper, easier and higher yield
  7. Use 0402/0603 passive components with enough space for easy Pick&Place
    1. 0201s are the size of a grain of sand – difficult to manage
    2. Larger sizes cost more with no significant benefit
  8. Minimize manual assembly, wiring & soldering
  9. PCB cost is driven by turnaround time
    1. Waiting for an extra week saves money

PCB Design Recommendations

  • Use the Mini Simplicity debug footprint
    • Avoid custom footprints!
      • Requires fragile & expensive jig for debug/production
    • Use the standard Tag-Connect footprint & cable
      • The spring post cable has alignment pins enabling reliable programming and test
    • Ideally make the footprint visible under a battery door to allow (re)programming after the device has been inserted into the enclosure.
  • No wires under the RF traces
    • Ideally all RF traces are all on component side
    • Minimize vias on RF traces which are impedance discontinuities
    • Surround RF traces with ground vias
  • Use the vendor recommended crystals as they have been vetted by the vendor
  • Continuous Ground fills
    • Ideally have a continuous ground fill on the back for better RF
  • Add thru-hole vias to the VCOM TX signal
    • See when printfs are coming out on a scope compared to current consumption
    • Can be done by making a via larger (1mm)
  • Connect extra GPIOs to thru-hole test points
    • Accessible from both sides of the PCB
    • Easily prototype hardware changes or future features
  • Add a hardware version GPIO Resistor Divider Pair or capacitor
    • High ohms so little currentVersion 1=pulled up, Version 2=pulled down
    • Version 3 onward uses the ADC to measure voltage or a cap and a timer
  • Review the ground plane flood fills
    • Ensure current paths do not require changing layers – No antennas – No floating areas
    • Array vias between top and bottom
  • Avoid wires under the Z-Wave chip
    • The EP prevents this anyway
  • Silkscreen signal names on connectors and test points
  • All ground connections must have thermal reliefs
    • WSTK has several pins flood filled to a thru-hole connector – nearly impossible to solder!
  • Analog has special rules! Follow them!
  • Have a checklist, follow it, add new items to avoid previous mistakes
    • DRC rules must pass
    • All nets routed

See my public ZRAD github repo for detailed procedures for going from schematics to PCB layout to assembling prototypes.

Next Steps

Part 7 of the Z-Wave Developer’s Journey discusses how to debug and optimize battery life. Matching the proper type of battery to the needs of your product and achieving the desired battery life requires classic engineering skills. I’ll go over a few tips from my years of optimizing coin cell battery devices to last for years. As we continue along the Z-Wave Developer’s Journey, I welcome your comments and questions.  Please feel free to reach out to me directly via email.

A Z-Wave Developer’s Journey | Part 5

DrZWave2 Comments

Z-Wave Firmware Hardening

How to avoid truck rolls though resilient firmware coding techniques

This is an abbreviated version of the full blog on the Z-Wave Alliance website.

Every IoT device has bugs. Today’s devices have many thousands of lines of code and too many hardware features meaning there are plenty of bugs hiding in every device. Bugs are in your product. I know it, and you know it. I propose that the best solution for these bugs is to “harden” your firmware to make it more resilient and keep on truckin’ if something bad happens. Below are my Tips and Techniques for hardening Z-Wave firmware to survive a failure. These ideas are for Silicon Labs SDK but similar techniques apply to Trident IoT.

Seven Tips to Harden Z-Wave Firmware

  1. Assume Everything is Broken
  2. Use FOR Instead of WHILE
  3. Replace Default_Handler
  4. Enable the Other Watchdog
  5. Reboot if no Comms in a Day
  6. Enable Stack Overflow Checking in FreeRTOS
  7. Run Static Analysis Tools

1. Assume Everything is Broken

This is a philosophical idea you need to keep in the back of your mind with every line of code you write. Assume everything is broken all the time – hardware never goes “ready”, a queue is always full, a mutex never switches, an impossible state occurs and similar sorts of failures. The most common code technique is to always check for error conditions of any function that returns a value. Always check inputs for validity.

The most insidious failures are stack overflows. This can happen where parts of RAM are overwritten and I’ve found it amazing that the code can keep running even after trashing potentially hundreds of memory locations. The challenge is there is no way to predict what might happen. All sorts of things that “can never happen” absolutely will happen when the stack overflows. Due to the limited RAM on Z-Wave chips, it is easy to overflow the stack.

Another common impossible condition is when the power supply sags just enough to flip bits in ways that are technically impossible. Strong magnetic fields from nearby motors and even cosmic radiation can flip bits in impossible ways. There is truly nothing that “can’t happen”. Thus, always code with the thought that the impossible can happen because eventually it will.

Wireless IoT devices using Z-Wave are often wired directly to mains-power. They cannot be easily rebooted like you do with your computer when it freezes. If a device bricks, it’s dead potentially for months or even years before a power failure brings it back online. Once a device is wired in, it’s usually there for many years and with Z-Wave it can be decades. Ensuring the firmware is resilient when (not if!) the impossible occurs will keep customers happy since they never knew the device rebooted – it kept on truckin’.

2. Use FOR instead of WHILE

The Silicon Labs SDK, including the bootloader, has many while(hardware_busy) loops that will wait forever and can cause the device to brick. For example, in Silicon Labs if you enable the LFXO (32KHz crystal oscillator) but don’t have the crystal wired up, the startup code waits for the LFXO to be “ready” with a while loop. While this is obvious when debugging firmware, this loop causes a device in the field to brick if for some reason the crystal stops working. The simple solution to this is to add a FOR loop with a timeout enabling the code to continue.

Example in em_cmu.c:

Replace: while ((LFXO->STATUS & _LFXO_STATUS_ENS_MASK) != 0U) {  }

With: for (int i=0; (i<1000)&&((LFXO->STATUS & _LFXO_STATUS_ENS_MASK) != 0U); i++) {__NOP()}

Note the __NOP() is necessary to prevent the compiler from optimizing the loop and removing it. The timeout value (1000 in this case) must be chosen based on testing. I usually set it to 10X the typical value. Following the FOR should be an assert to check that the timeout didn’t occur. Note the use of < and not == for the check of the timeout. If the impossible were to happen, there is a chance “i” could skip past exactly 1000 and then since this is a 32-bit number, the timeout would be waiting for a long time for the 32-bit number to wrap all the way around. This is another defensive coding technique where in the back of my mind I’m thinking of the impossible and coding to be resilient even when the impossible happens.

3. Default_Handler

Segger has a great article on debugging the many “fault handlers” in the Cortex-M processors. The article provides code for many different handlers to help debug the fault and make the code more resilient.

Default_Handler is in startup_<chipnumber>.c and is unfortunately NOT declared as weak so you must edit the SDK file itself. Best to select the Copy Contents mode in the .slcp file Import Mode. Maybe it’s better to fix in the SDK for all your projects! The Silicon Labs SDK has only a single line while (true); for a default handler. This code counts on the watchdog to eventually reboot the chip. But the reboot isn’t guaranteed and there is no additional debugging information as shown in the Segger examples. All the fault handlers are mapped into this single handler, but they can be individually overridden due to weak assignments. Even something as simple as a divide by zero can cause a fault handler to be called and brick the device.

At a minimum, put the Segger recommended code in for at least some of the exception handlers to make debug easier. Generally, it is a good idea to light an LED (ideally red) or some other external indicator to help during debug. Other ideas are to log the address and condition that caused the exception and store it in the User Data Page/NVM that can be read out from production units that were returned from the field by angry customers. Then perform forensic analysis to identify the cause and release a firmware update that solves the problem.

4. Watchdogs

Watchdog timers are crucial for reliable 24x7x365 operation of an IoT device. A watchdog timer is a timer that slowly counts down. Every now and then, the firmware “feeds” the watchdog by resetting the counter to a high value. If the counter reaches zero, a full reset of the chip is triggered which reboots the chip and hopefully resolves the error condition. The watchdog timer typically takes a couple of seconds of being starved before the reset to ensure it doesn’t falsely reset. The trick to a resilient watchdog is deciding when to feed it, and more importantly, when not to. I wrote a blog post on watchdog timer best practices back in the 500 series days which still applies.

5. Reboot When no Communication for a Day

Hello, is anyone listening? The concept here is basically a long-duration watchdog timer. If the controller hasn’t sent a frame and/or hasn’t acknowledged the receipt of a frame in twenty-four hours, maybe a reboot will clear things up. I’ve seen this in the 500 series where on rare occasions reading the HomeID from the external NVM would fail. As a result, the device would forget the HomeID and assume some random number. This random number would then be stuck in the device for days or weeks or even years until the device rebooted for some reason. This was a classic Impossible Condition that seemed to happen on a fairly regular basis when several tens of thousands of Z-Wave devices have been operating for a few months. The only solution for the end-customer was to rip the bricked device out of the wall! Or more commonly factory reset it and rejoin the network which resulted in one-star reviews.

The solution is simple, setup a 24-hour software timer and check the RX/TX statistics and if nothing has made it through, reboot! Rare impossible conditions are fixed in a way that customers never notice. This check is only needed for always-on or FLiRs (LSEN) devices as deep sleeping devices reboot every time they wake up.

6. Stack Overflow Checking

A real-time-operating-system adds complexity, but FreeRTOS has a feature to check for a stack overflow which is enabled by default. The variable configCHECK_FOR_STACK_OVERFLOW is set to 2 by default in FreeRTOSConfig.h. This enables some checking and fills the stack space with 0xA5s which is then checked with each task switch. Inspecting RAM after running the code for some time in the debugger can provide insights as to how close to overflowing the stack has happened so far. The check calls vApplicationStackOverflowHook if there is a failure but there is only an assert in the weak function. My recommendation is to add a breakpoint here during testing and consider rebooting in the released code. Stack overflow checking is only recommended during development and testing due to the additional overhead.

The insidious problem with stack overflows is they often require several things to go wrong at the same time – a task switch, an interrupt, the radio sending or receiving data and having to find a new mesh route, code allocating sizable temporary buffers and maybe even more code that uses up the limited stack space. As mentioned above, I have observed the stack overflowing, trashing many dozens of memory locations and the code keeps running but eventually there is an impossible condition or more often a hardfault exception. As a result, the failure is often overlooked as it only happened that “one time” but in reality, it happens a lot. Getting the failure to happen repeatedly in a controlled environment is often very difficult. I have had dozens of units set up testing a specific failure case which would take all weekend to finally trigger. Then not having enough data on the unit that failed makes it even more exasperating.

7. Static Code Analysis

Use Claude, CodeX or other static code analysis tools to review all firmware. AI continues to improve at an exponential rate to grade code quality. Often AI can recommend changes to fix the code, but I would carefully check over the suggestions. AI can hallucinate or simply start making things up out of nowhere. The GCC compiler has a -fanalyzer option that will find a few interesting things. Coverity is an industry leader in this field but is pricey. What tools have you used?

Next Steps

Part 6 of the Z-Wave Developer’s Journey discusses hardware best practices. I present my tips and tricks for making low-cost, easy to debug and manufacture Z-Wave products from my 25 plus years of Z-Wave experience. As we continue along the Z-Wave Developer’s Journey, I welcome your comments and questions.  Please feel free to reach out to me directly via email.

A Z-Wave Developer’s Journey | Part 4

DrZWaveLeave a comment

Coding and Debugging Z-Wave Firmware

This is an abbreviated version of the full blog on the Z-Wave Alliance website.

Each of the silicon vendors, Silicon Labs and Trident IoT, have their own coding and debugging tools and methods. Each vendor has training on their tools which will help you up the learning curve. In this part of the journey, I present my tips and hints to help you up that steep learning curve. The first step is to watch the training videos or read the getting started guides from each vendor. They are well worth your time. While the videos are not particularly entertaining, you can skip through some parts and watch them at 1.5x which is what I did.

Silicon Labs Simplicity Studio 6

Moving from Simplicity Studio 5 to version 6 requires significant learning as the Integrated Development Environment (IDE) is now based on Microsoft’s Visual Studio Code instead of eclipse. While some parts are familiar, the IDE is markedly different and will take some getting used to. Simplicity Studio 6 (SSv6) is new and has a few wrinkles and rough spots that Silicon Labs will be smoothing out in the coming months. Don’t hesitate to file a case on the support portal or contact your Silabs FAE for help. The first step is to install SSv6 and Visual Studio (VS) Code and the Silabs extension for VS Code. Open SSv6 and you get a broad view of all the wireless protocols Silabs supports, click on Z-Wave to open the list of sample applications. Pick a sample application such as Switch On/Off which is recommended to start with. I recommend using the Copy Contents option. Using the Link SDK and Copy Project Sources option limits VS Code’s ability to find and search through files. The ProjectName.slcp file will open which is where most configuration takes place. The Software Components tab is the doorway to installing command classes or peripheral drivers. Search then install and configure command classes or APIs as needed for your application. The Pin Tool shows a graphical and tabular representation of the GPIOs. The .slpb file is the post-build editor which is where the scripts for keys and OTA file creation are configured but I recommend leaving those at their defaults. Now for the big step – in the upper right corner is the Open in VS Code. Click on that to open the Microsoft tool.

My tips and tricks for using VS Code:

  • Always press to save after editing a file!
    • SSv5 would automatically save when you click on Build, VS Code does NOT
    • It has taken me weeks for this to become a habit. I have been debugging for an hour only to realize the change I made is not in the download because I didn’t hit save
  • Copilot AI is a huge productivity improvement – learn it – use it!
    • Add a comment to any line of code by typing “//” at the end, Copilot usually produces a decent, often wordy, comment on what the code is doing
    • Start typing any function, Copilot fills in the required parameters in the right format(s)
    • Start typing “for”, “while” or “switch”, Copilot fills in much of a likely block of code and as you fill in more code, Copilot guesses what else you need – press TAB to accept the suggestion
    • Copilot hallucinates – it is not perfect so be careful and obviously check the results
  • Searching for variables or functions is different in VS Code than SSv5
    • Hovering will display some information, ask Copilot for more details
    • Right Click opens a popup with many options – recommend Find All References
    • Use the Search menu on the left instead of the bar across the top
      • In Files to Include – add “*.c,*.h” otherwise the search will include object files, .map files and other useless hits
      • Note that the search is across your entire workspace so look carefully at which project the search result is in
    • I sometimes still end up greping to find things
  • Can’t see a file in the Si Extension? Use the Explorer tab (upper left) to see all the files
  • Clean and then Build from scratch if something is not working
  • From the Silabs Extension you can flash firmware, open a terminal, open commander, and other tools
  • The gnudbg debugger is reliable
    • but I still use Segger Ozone when I hit a really squirrely problem
  • Get a new computer with at least 32GB of RAM
    • SSv6 and VSCode are memory hogs
    • I bought a new laptop with an i9 CPU and 32GB which made the experience more efficient and stable

Trident IoT

Trident takes a quite different approach to embedded programming than Silicon Labs. Building a project, compiling and flashing are done using their command line tool called “Elcap”. Elcap relies on Docker or Podman containers to make for a unified development environment regardless of the platform: Windows, Linux or Mac. Trident has a VS Code plugin called “TIDE” which makes for a familiar environment for writing and exploring the embedded code. These features make for frictionless setup as the environment is encapsulated within the container. The container environment also makes support easier as the entire environment can be recreated months or years later and reproduce the identical firmware download. They also claim that upgrading to newer SDK versions will be possible with a single command line. Silabs developers usually must rebuild their entire application starting from a fresh sample application in a newer SDK to upgrade.

Trident IoT Tips and Tricks:

  • Use the latest version of elcap – elcap about
  • elcap self doctor – checks your environment, execute this if something is not working
    • The problem is you usually need to start the Docker Desktop application
  • Append –help to any elcap command for more information
  • Trident relies on the Segger Ozone Debugger
    • VS Code integration with Ozone is expected in a future release. elcap creates a Segger .jdebug file for Ozone as part of the build process
    • In config.cmake set the following so Ozone finds all source code
      • Set(ZWDSK_CONFIG_USE_SOURCES “ON”)
      • Takes a little longer to compile but can single step ALL the code
      • Turn it off again when close to release for faster compile times
  • Use a Segger J-Link to debug your own PCB
    • The connector pinout is NOT the same as the Silabs MiniSimplicity header. The Trident devkit has a J-Link built in but does not debug external PCBs
    • The DKR-HOST can be used to debug trident based PCBs
  • The build environment is based on CMake
    • To configure the project, edit:
      • CMakeLists.txt
      • App/CMakeLists.txt
      • Configuration will be simplified in a future release
  • Complete reference firmware is available on request for several sensor types
  • Start the PCC, Zniffer or Tridents own cross-platform Z-Wave/Zigbee Sniffer from elcap

Next Steps

Part 5 of the Z-Wave Developer’s Journey discusses an often-overlooked schedule hit that in my view is absolutely critical, Firmware Hardening. Developers must keep in mind that IoT devices are often wired to mains power and may run for years without ever rebooting. Thus, the firmware must be resilient to account for impossible conditions that should never happen, but in the real world they often do. Remaining connected to the Z-Wave network and never “bricking” is critical to the success of your product and the reputation of Z-Wave. Do you have some Tips and Tricks to share? Please reach out to me directly via email.

A Z-Wave Developer’s Journey | Part 3

DrZWaveLeave a comment

This is an abbreviated version of Part 3, the full article is on the Z-Wave Alliance Web site.

Which Z-Wave Command Classes to Use for Your Z-Wave IoT Device and Why

Command Classes are the key to Z-Wave’s application-level interoperability. All protocols have a standardized physical layer which ensures devices manufactured by different companies can communicate. Z-Wave’s physical layer is defined in the ITU-T G.9959 standard. The physical layer standard is necessary, but insufficient for IoT devices to communicate in a meaningful way. Command Classes are the key to enabling a controller to “know” how to turn on a light when motion is detected and adjust the thermostat to the liking of the user. Command Classes are defined in the Application Work Group Z-Wave Specification available on the Z-Wave Alliance web site.

The “spec” is over 1300 pages long. If you need help sleeping one night, crack this open and you’ll be off to lala land in no time! Fortunately, it is not a document you read from cover to cover. It is more like a dictionary, where you look up specific items to understand exactly how an IoT device communicates in an interoperable way, so all parties properly communicate the information. I make extensive use of the bookmarks bar in the PDF reader to quickly jump to the section I need. Search is also valuable to find the answer to a specific question.

Z-Wave Application Work Group Specification Z-Wave Alliance

When you first open the specification, you’ll see many versions of a command class and numerous ones that are obsolete or deprecated. If a command class is obsolete, you cannot use it ever and as a controller you don’t have to support it. Deprecated command classes are mostly old versions, and you must use the newer ones. Note that it is NOT required to support the latest version of a command class. For example, Battery Command Class version 1 is perfectly fine for most devices. Versions 2 and 3 add extra information for special types of batteries, but version 1 is fine for most IoT devices with simple batteries.

Another key Z-Wave file is the ZW_classcmd.h file which is in the SDK. This file explicitly defines every field of every command in every command class. The spec gets you to the right command, but the ZW_classcmd.h file defines the exact syntax and spelling of the fields you need to put in your code. Fortunately, VS Code does a pretty good job of filling most of this in for you when using Simplicity Studio V6. I’ll go into more coding details in the next blog post.

Where Do I Start?

What’s the first step in coding a new Z-Wave Product? You’ve already chosen the sample app in the 2nd blog and that choice goes a long way toward your first step here. Start with the Device Type V2 Specification section 7. There is a long list of common devices like switches, locks, bulbs, thermostats, sensors, and gateways. Your device should fit into one of these broad categories. Each Device Type specifies a list of mandatory command classes.

Mandatory Command Classes

The sample app typically provides all the mandatory command classes so there is no work required here. But you should double check as requirements change and sometimes the code lags the specification. The Z-Wave Certification Test Tool will do a comprehensive check for all the mandatory command classes so it’s worth a few minutes to check early in the project development.

Mandatory command classes are mandatory for a reason! They significantly improve interoperability! They help make most devices operate in a predictable way and provide similar information. They also ensure your device can be probed for all salient features enabling the hub to offer all your features to the user. This allows your product to be supported the day it starts shipping without waiting for the hub vendor to “support” your product thru manual coding. Note that section 7.2 of the specification has a list of mandatory command classes that are required for all products. These include Z-Wave Plus Info which helps the hub know exactly the general type of device, Association which tells the device where to send unsolicited reports, Firmware Update which enables updates in the field which is now required for the new global security initiatives.

What to do for Command Classes That Haven’t Been Implemented Yet

Not all command classes have been implemented yet. Can AI write them? Probably – let me know if you can create a working command class with AI. None of the thermostat related command classes are in the open-source repository as mentioned above. What do you do? You must implement them yourself. Ideally you should submit your implementation to the open-source repository to help the entire community. This is a pretty high-bar as you must implement every command in the command class and follow the coding rules as well as provide test code to ensure the code is bug-free (maybe bug-lite?). I’m currently implementing Geographic Location and Time command classes which I hope to Pull Request into the repository in the coming months.

The key is to copy a similar command class to use as a starting point. The first step is to implement the REGISTER_CC_V6 macro. Each existing command class has this macro, or an earlier version of it, at the bottom of the main command class file. The macro installs the command class into the Node Information Frame (NIF) and provides links to the command class handlers and other functions. The NIF is the list of command classes the device supports which the hub uses to interrogate the device to learn what it can do when first joined to a network. Next, implement the handlers that decode all the commands of a command class and return a report when a Get command is received. The command to send a frame is zaf_transport_tx() which puts the message into the FreeRTOS queue and sends it to the SDK. A callback function is called when the message is sent with a status of success or not.

Next Steps

Part 4 of the Developer’s Journey will discuss details about coding and debugging Z-Wave firmware. This is a longer blog as there is a lot to cover. I will include several screen shots and recommend tools that I find invaluable. As we continue along the Z-Wave Developer’s Journey, I welcome your comments and questions.  Please feel free to reach out to me directly via email.

A Z-Wave Developer’s Journey | Part 2

DrZWaveLeave a comment

This is an abbreviated version of Part 2, the full article is on the Z-Wave Alliance Web site.

Introduction

How do you get started in developing a wireless IoT product using Z-Wave? Assuming you’ve chosen a silicon vendor from Part 1 of this blog, the next step is to become familiar with the tools, developer kits and software of the respective vendors, Silicon Labs and Trident IoT. Both vendors utilize the popular Microsoft Visual Studio (VS) Code Integrated Debug Environment (IDE). Each has developed an extension to customize VS Code for their respective SDK. If you’re not already a VS Code user, you should be. The Intellisense AI feature is a game changer for managing the large amount of code in the SDK you will be interfacing with for your project. I am relatively new to VS Code and I am still in the learning phase. Please comment on this blog If you know of any time saving tricks that I’ll be happy to pass on to the rest of the community.

I highly recommend taking the training and reading documentation from each vendor on using their tools and developers kits. In later postings I’ll be using these tools and assume you are already familiar with them.

Software Architecture, FreeRTOS and the SDK

Fundamentally the Z-Wave SDK relies on the open-source FreeRTOS real time operating system. FreeRTOS provides many resources such as multitasking, software timers, memory management and security.  The Z-Wave SDK is in one task, your application code is in another task and then there are a few utility tasks. The use of an RTOS makes the code more modular but also more complex. Instead of simply calling a function to send a message over the radio, the application task sends the message through a queue to the Z-Wave task which then sends it over the radio and later returns the result to a callback function you passed through the queue. When the RTOS determines there’s nothing to do, it will put the chip to sleep. An always-on device will only put the CPU to sleep and leave the radio on, but battery powered devices will go into a low power mode.

Each vendor has some amount of the SDK pre-compiled into a library. Mostly this provides an abstraction layer that gives the vendor a level of intellectual property protection. Much of the code is in source code form and you will compile the SDK with your code as well. Trident has a method to compile all the source code into your project which can make debugging the SDK possible. The SDK includes lots of helper APIs and code for many common command classes. If the command class you need is not available (yet), you will want to copy the code of a similar command class. To be efficient, you need to reuse as much code as possible. Gotta love copy and paste.

What to Customize Next

Below is a list of things I customize for any new project. I’m using the Silicon Labs SDK in this case but Trident is similar and starts by editing the app/CMakeLists.txt file. Open the .slcp file in Simplicity Studio V6, click on Software Components, then Installed, then open the Z-Wave list.

  1. Z-Wave Core Component – Select the Z-Wave Region to match your location
    1. Max Tx Power will need customization when preparing for regulatory approval
  2. Z-Wave Version Numbers – Turn on (True) Use Application Version and enter version numbers
    1. The Minor Version MUST be incremented for OTA firmware update
  3. Z-Wave ZAF Component – Several items must be customized for your product
    1. The Manufacturer Specific ID, Product Type and Product ID are a 48-bit unique identifier for the product – basically a fingerprint
  4. Command Classes – Association CC – Recommend a single Lifeline NodeID
  5. Uncheck Installed – then install Z-Wave Debug
    1. This will uninstall Z-Wave Release which uses higher compiler optimization and the debugger is unable to accurately single step C source code
    1. Note that OTA fails when DEBUG is enabled due to code size with the lower optimization
  6. Z-Wave Log – optionally turn on more logging which will print more messages out the UART
    1. Entering vcom into all 4 debug levels will print a lot of messages
    1. Be sure to turn this OFF when getting close to a release
    1. Note that the sending text out the UART is a blocking operation and will change how the code runs and may cause a watchdog reset

These customizations are just the start! From here you will install other command classes, SPI, I2C or UART drivers and of course your own custom code.

See more details at the full blog posting on the Alliance Web Site via this link: https://z-wavealliance.org/a-z-wave-developers-journey-part-2-2/

Please comment below on this or any of the topics in the Developers Journey.

Join me at the Z-Wave Summit May 27-29 in Vienna Austria. Unplug Fest is the afternoon of the 27th which I will be coordinating. We’re not doing range testing this time around but will be demonstrating some of the main features of Z-Wave including SmartStart, Multicast, Mesh routing and more. Bring your new devices and test them with several ecosystem players to see Z-Wave interoperability in the real world.

A Z-Wave Developers Journey | Part 1

DrZWave1 Comment

The Z-Wave Alliance is funding my writing of a blog that describes how to develop a Z-Wave product. The “Journey” is a series of ten blog postings with step-by-step descriptions of how to develop a Z-Wave product from idea to volume production. The full blog posting is on the Alliance web site but here is an abbreviated version.

Introduction

A Z-Wave Developer’s Journey is a series of ten blogs on the nuts and bolts of creating and bringing to market a wireless IoT product utilizing Z-Wave. The series provides a step-by-step roadmap for an engineering team to bring their idea from the concept to a product ready for volume manufacturing. Naturally, this series can’t delve into every aspect of the process but leverages vendor training, documentation and Github to flesh out the details. The journey focuses on Z-Wave end devices but a similar process would be followed by Z-Wave controllers. One thing to note is that everything is constantly changing. The Z-Wave specification continues to evolve with new Command Classes and updates to existing ones, the vendor Software Development Kits (SDKs) have new releases every few months and new silicon chips are always being released. While the guidance shared here is relevant today, details will inevitably evolve over time, so stay engaged and enjoy the ride.

Topics

The journey begins with this blog which describes the topics to be discussed in this ten-part series. You have the opportunity to comment on these topics as each is published. Feel free to comment or reach out to me directly at DrZWave@DrZWave.blog. I continue to learn by doing and enjoy exchanging best-in-class techniques for IoT product development of both hardware and software even in “retirement”. Below is a list of planned topics though the list may morph somewhat along the way based on your feedback. Don’t be shy, comment below or send me an email.

  1. Introduction & Z-Wave Silicon Choices
  2. First Steps in Customizing Z-Wave Firmware
  3. Which Z-Wave Command Classes to Include and Why
  4. Coding and Debugging Z-Wave Firmware
  5. Firmware Hardening
  6. Z-Wave Hardware Design Best Practices
  7. Optimizing Battery Life
  8. Antennas for Z-Wave
  9. Z-Wave Regulatory Process
  10. Z-Wave Volume Manufacturing

See the Alliance blog posting for details on the available Z-Wave silicon.

How to Choose

Which Z-Wave chip should you use for your project? Of course, the answer is… depends. The main challenge with the ZG23 is the limited amount of flash and RAM. The SDK uses virtually all the available resources. If your product is fairly simple, like a door/window sensor, the ZG23 should be fine. If you are designing a thermostat or door lock, I would recommend either the ZG28 or the CZ20. If you use the Silicon Labs QFN48 you can develop using the ZG28 and then potentially reduce cost by switching to the pin compatible ZG23 if the code fits. The ZG23 could also work out if you connect an external serial flash chip for the OTA image. That frees up half of the 512KB of flash for your application but it’s still tight on RAM. The ZGM230 module is easier to manufacture since the crystal is calibrated at the factory but is limited to +14dBm transmit power thus effectively cutting the RF range in half. The choice of Silicon Labs or Trident IoT is a more nuanced choice based on the support and relationship you have with the vendor.

Feel free to comment below or contact me with your thoughts or topics you need answers!

Save the DATE! EMEA Z-Wave Unplug Fest and Summit Vienna, Austria, May 27-29.

Unplugfest 2025 Know Before You GO!

DrZWaveLeave a comment

Z-Wave UnplugFest 2025 will be held at the Lake Elsinore Skydive center in California October 28, 2025. Here are a few things you need to know before heading out to test your Z-Wave device and your BRAVERY!

Travel Plan for Z-Wave Unplugfest 2025 in Lake Elsinore California

Before You Head to Unplugfest

  1. IF YOU HAVE A Battery Powered End Device (FLiRS or Sleeping EM4):
    1. Your End Device MUST have the Listening Bit Set and thus be an Always-On device
      1. Temporarily alter your firmware to be Always-On. In SSv5 open the <project>.SLCP file, Click on Software Components, Scroll down to Z-Wave, Click on Role Types, Install Z-Wave Role Type Always On, You can switch it back later or make a copy of your project specifically for Unplugfest
      We will be using ZRADmini boards with a GPS receiver to track the exact location during Z-Wave Long Range testing. Your device does NOT have to support Geographic Location V2. Your device will be one of up to 2 other devices that will be tracked using a GPS enabled ZRADmini during RF Range Testing. However, for the maximum accuracy of RF Range testing you can add GeoLocV2 with a GPS module to your end device
    1. If your End Device can be powered via USB-C, we will have that covered, otherwise bring sufficient batteries for several hours of continuous operation
  2. Your End Device must have a visible QR code and PIN on it – all devices will be joined with S2 Authenticated or Access keys
  3. Bring your own source of POWER – Especially for Controllers
    1. We hope to have limited amount of 120VAC power from the Skydive center BUT maybe not!
    1. Better to bring your own battery inverter like a Jackery
  4. Are you brave enough to go skydiving?
    1. Signup here – Recommend you book online ahead of time!
  5. Arrive by Monday – the bus LEAVES at 9:00am SHARP Tuesday morning on a 75 minute drive
  6. Bring extra units, WSTK to reprogram units, firmware project, extra cables, batteries, tape, etc.
    1. Plan on things failing so bring extras! Unplugfest always seems to follow Murphys Law

Timeline for the 2025 Z-Wave UnplugFest

  • 8:00am – Gather at hotel Cassara lobby and board the bus at 8:45
  • 9am SHARP – bus leaves for Elsinore airport – 1+hr
  • 10:30 – setup at end of Cereal St or at Skydive Elsinore
  • 11am-3pm – RF Range Testing
    • If weather permits, JonLuke West will skydive while tracking altitude via GPS
      • If you’re brave enough, join him on a tandem jump via Skydive Elsinore!
        • Must be booked in advance which YOU must call and arrange yourself!
      • Or soar in a 17m sailplane however the plane is metal which may block RF
    • Eric will have one ZRAD as a Controller and another as an End Device with Geographic Location V2 and a GPS receiver to track the exact locations of up to 3 additional devices at a time
      • DUTs can run standard firmware (without needing GeoLocV2) and will confirm their range using fully secure Get/Report while obtaining the GPS coordinates from ZRAD
      • All devices must be joined with S2 Authenticated Security enabled and US Long Range
  • Noon Box Lunch for all who registered for Unplugfest
  • 3:00pm Testing Ends – Board Bus
  • 3:15pm SHARP – Bus departs
  • Wrap Up discussion – 3:30pm-4:30pm on the bus
    • Discussion with all attendees on findings and general discussion of Unplugfest, Z-Wave, development tools, the market, the latest football scores, and the best beer in Carlsbad
  • The Z-Wave Summit kicks off in the hotel at 5:30pm

Check out the video from the 2024 Unplugfest in Austin TX: ZWA2024AustinUnPlugFest.mp4 and DrZWave’s blog post: https://drzwave.blog/2024/05/08/z-wave-summit-2024-austin-tx/ and JonLukes 2.7mi Z-Wave Skydive: https://youtu.be/lcuyDC128cQ?si=09wVd3cDV89326wW

DEBUGPRINT Is Now ZPAL_LOG

DrZWaveLeave a comment

The 2025.6 Silicon Labs SSDK Z-Wave release changes the familiar DEBUGPRINT utility to ZPAL_LOG. A brief mention of the change is in the Important Changes for the Z-Wave 7.24.0 release however I’ll give a more detailed description of what has changed and how to use this new utility. If you’re trying to upgrade a project from an earlier SDK, this “breaking change” will make that effort more difficult as you have to switch any DPRINT statements to the new ZPAL_LOG format.

DEBUGPRINT

First a little history. The DEBUGPRINT utility came from the original early 2000s Zensys code running on the 8051 8-bit CPU. From the initial 100 series thru the 500 series, DEBUGPRINT was the ONLY debug utility we had! Even though this was the 2000s thru most of the 2010s, debugging using PRINTF was obsolete by the late 1980s – but not for Z-Wave. PRINTF would print out a short message and maybe a hex value or two and we would try to divine what was going on from these tiny details. There were no breakpoints, no ability to single step the CPU or watch RAM or registers change as the code was executing. Because sending characters out the UART would block the CPU until the UART was empty, you could only send a few characters and maybe a hex value. Sending more characters out the UART would alter the flow of the program and either mask the bug or break something else or often cause a watchdog reset! UGH. While primitive, it was all we had and we made do. DEBUGPRINT was easy to enable or disable – just #define DEBUGPRINT in the file you needed the print statements and recompile. Typically, only a few print statements were enabled at a time or the UART would spew out too much detail or messages would get mixed together making gibberish of the output.

Debugging Today

Debug is much easier with the modern ARM CM33 processor on todays Z-Wave chips. We have Single-Wire-Debug (SWD) and can set breakpoints, watchpoints, single step, view memory and peripherals. There’s even Trace where every instruction executed is logged enabling the precise flow of a program to be viewed after the trigger condition of a bug. We still have DEBUGPRINT which is handy for printing out high level messages like “eReset Reason=3′ each time the chip reboots. These messages are handy during debug and were easy to remove prior to building the final production code simply by commenting out the #define.

ZPAL_LOG

The Z-Wave Log component in a Simplicity Studio project .slcp file is installed by default in all the sample applications starting with the 7.24 release. This is good thing and a bad thing. It’s good since it’s already installed and printing messages before you’ve started modifying the code for your project. It’s bad because when you should remove this code for the final production build, we’ll get into that shortly.

General

When you Configure the Z-Wave Log component you get a lot of radio buttons to switch various features on or off. The first block “General” includes:

  • Size of the buffer (default is 96 bytes) – Leave this as-is
  • Display log level – prints 3 characters identifying Info, Debug, Warnings or Error messages
  • Display log source component – prints a number which identifes which module the print statement is from
  • Display log timestamp – prints a decimal number of ticks since reset

Enabling any of these messages slightly increases the total FLASH usage. The bigger problem is that it increases the number of characters sent out the UART potentially causing the CPU to block and thus changing the program flow. I recommend leaving these at their defaults.

The Display Log Source component would be useful IF SSv5 decoded the numeric value for you and displayed the name in the console window. Unfortunately it just prints a number which you then must manually decode by looking up the value in the enumerated type zpal_log_component in the zpal_log.h file. The trick would be to send just the numeric number out the UART (thus not taking up significant amounts of time or FLASH) and then SSv5 expanding it to the component name. I have suggested this to the SSv5 team in the past but hasn’t yet come to fruition. The source component is useful if there are several similar or even identical PRINT statements in different blocks of code to help identify where it is coming from. Also if you enable a lot of PRINT statements, it can help narrow down which block a specific message is coming from.

If you enable all 3 of these switches you get: “000000023 [I] (1) ApplicationInit eResetReason = 7”. Where the 00000023 is the timestamp; [i] is the Log Level (Info in this case) and (1) is the component which you have to then decode yourself as mentioned above (ZPAL_LOG_APP in this case) .

Output Channels

The next block is the Output Channels which can enable the four levels of messages: Debug, Info, Warning and Error messages. The default has just Info enabled with the word “vcom” entered into the Info box. I recommend enabling all four of these by entering “vcom” into all four of them. This will increase FLASH usage as the strings for all these messages are now stored on-chip.

Component Filtering

The final configuration block in the Z-Wave Log component is the Component Filtering section. This section has a bunch of radio buttons you can enable which will turn on various messages from the respective module. If you enable ALL of these, it’ll add about 10K bytes of FLASH so maybe only enable ones you think you need.

UART Blocking

The EUSART is only running at 115200 baud which makes each character take over 10 microseconds to send out the UART. This might seem fast, but when you’re transmitting long strings of characters this can seriously impact the program flow in real time. If the UART buffer fills, the code will stall and wait for more buffer space to fill (see the “while” loop in eusart_tx in sl_iostream_eusart.c). If you’re spitting out a lot of messages, the UART will fill causing the program flow to change potentially masking the bug or sending you down the wrong rathole. The baud rate is easily increased to 1mbps which will speed up the printing by 10X which helps, but you need to be aware of the amount of messages pouring out the limited speed of the UART.

Disabling ZPAL_LOG

Once debugging is complete, the next step is to compile all of these messages out of the production code. Fortunately this is easy – just remove “vcom” from all four of the Output Channels section of the Z-Wave Log component. While this removes most of the code for debug messages, the Command Line Interface (CLI) is still there and the EUSART is still used by the sample applications. When developing your own application you will typically want to remove ZPAL_LOG and the CLI as it’s not something your application needs – it is only for the Silicon Labs demos.

Conclusion

Why did silabs make this “breaking change”? I don’t have specific insights from Silicon Labs but the new format provides easily configured “levels” of logging. Is that worth a “breaking change”? I wish they’d spend more time testing the code to reduce the bugs than making a lot of file changes tweaking features that don’t really improve the developers experience. If you’re going to change an important feature, it’s also important to document it which is sorely lacking. But I hope this post helps you utilize this new feature and not be quite so frustrated when updating your project to the latest SDK.

Dreaded Flash Memory Overflow Solutions

DrZWaveLeave a comment

The EFR32ZG23 Z-Wave 800 series MCU has limited FLASH and RAM available for the application. The 800 series actually has less FLASH than the 700 series which stored the bootloader in a dedicated 16K memory. Worse, the bootloader in the 800 series has grown from 16K to 24K! Features are always being added to the SDK making it ever larger leaving less for the application. Seems like Silicon Labs needs to spend some time squeezing the code instead of constantly adding features.

Here is the typical error message when FLASH overflows:

Description	
FLASH memory overflowed !	
make: *** [makefile:114: all] Error 2
make[1]: *** [makefile:123: SwOnOff_2024120_ZG23B_GeoLoc.axf] Error 1
region `FLASH' overflowed by 92 bytes
SwOnOff_2024120_ZG23B_GeoLoc.axf section `.nvm' will not fit in region `FLASH'

We can’t create more FLASH on the chip, it has what it has. But, there’s always software we can change! By default, the project has ZAF->Z-Wave Release installed which sets the C compiler optimization to -Os which optimizes for size which is probably what we want since we’re out of FLASH. However, deep in the configuration files there is the definition for SL_BOOTLOADER_STORAGE_SIZE which changes from 196K to 180Kbytes when NDEBUG is defined. NDEBUG is defined when the Z-Wave->ZAF->Z-Wave Debug component is installed. The question of why BOOTLOADER size is being reduced by only 16K when debug is enabled is unclear to me. However, in my testing, adding the DEBUG component still results in FLASH overflowing but now by 4344 bytes! Obviously the change in Optimization from -Os to -Og (debugging) blew up the code which is expected. I enabled DEBUGPRINT to get debugging information out the UART which increased the flash usage even more.

Since I am debugging and will not be testing OTA at this stage, I don’t care how big the bootloader storage size is since I am not using it. I need more FLASH space for debugging! Simply edit the sl_storage_config.h file and change SL_BOOTLOADER_STORAGE_SIZE from 0x2C000 to 0x20000 to free up another 48K bytes:

// <o SL_BOOTLOADER_STORAGE_SIZE> Size of the bootloader storage.
// <i> Default: 0x20000
// <i> Note that this value is only being used if BOOTLOADER_STORAGE_USE_DEFAULT
// <i> is set to false. This value will control how much of the flash memory
// <i> is reserved for bootloader storage.
#if defined(NDEBUG)
#define SL_BOOTLOADER_STORAGE_SIZE  0x00030000
#else /* defined(NDEBUG) */
//#define SL_BOOTLOADER_STORAGE_SIZE  0x0002C000    - original value
#define SL_BOOTLOADER_STORAGE_SIZE  0x00020000
#endif /* defined(NDEBUG) */

Now the project fits comfortably in FLASH with plenty of left over space. However, I cannot OTA it and definitely cannot ship it in this way for production. Once I’m done debugging, I’ll have to revert back to RELEASE mode and remove DEBUGPRINT. If FLASH is overflowing that will require some additional effort to squeeze back into the available space. I would first try Link-Time-Optimization (-flto) to the C compiler but that can introduce some instability and require significant amounts of testing time. Next, try looking for code you don’t need and remove it. After that, complain to Silicon Labs they need to shrink their code! 

Ram usage       :    65532 /    65532 B (100.00 %)
Flash usage     :   446868 /   491520 B ( 90.92 %)

RAM usage is at 100% is OK because the HEAP is expanded to fill the available space. But there is very little left over for the application as any RAM usage is making the HEAP smaller. The HEAP is used for all sorts of things like temporary variables, buffers and FreeRTOS. I am very concerned that some of the bugs in Z-Wave products are due to heap overflows. Heap overflows are very difficult to reproduce and debug as they typically require several failures to happen at just the right time. Unfortunately these failures seem to happen with regularity in the real world.

Hope this helps you get back to debugging quickly. Leave me a comment below with your helpful hints that I can include in a future post.