Engineers Must Learn the Importance of Software Testing

Posted by: Aaron Campbell

I started working at Datalight as a file system engineer shortly after graduating from college. And when I started, software testing was somewhat foreign to me. I was a developer, and QA was a place for other people that did other things. I assumed my job was to create software that performed as users expected as long as the users did things correctly. If they did something weird, like trying to fread() from a NULL file handle, they should expect bad things to happen. In other words, I expected to follow my “common sense” in designing workable software, and I expected my users to follow the same “common sense” as they interacted with it. Now, I’m not saying I argued this point out loud; it was more of a subconscious assumption. In college classes, this worked okay, but it did not produce the quality software that is needed for professional products, especially in the embedded and IoT space. For one thing, I came to realize my “common sense” was different from that of my users. Through my work at Datalight, I quickly learned the importance of gracefully handling all possible uses of the software—not just the common or expected cases.

As a vender of file system and flash drivers that often get built into things like airplanes and automobiles, Datalight doesn’t just test software before shipping. We also provide customers with extensive test suites (also in source code) so they can more easily verify that our software runs flawlessly on the target hardware. If a failure is ever exposed, our support team investigates the issue, and quickly alerts our engineers if they can’t find a clear explanation. It’s amazing that all of this happens in the same building; we often pull support, QA and engineers into the same meetings to address how we can exceed our customers’ goals. As I learned from seasoned experts* on each of these teams, I tried to pick up on the taste for quality and the high standards that are set for Datalight software.

Some of my developer friends may find it unprofessional to rely on intuition instead of using thorough planning, documentation and testing to develop software. However, every developer would rather spend their time coding than doing those other things. This becomes interesting as it relates to free and open source software. Free open source software has the advantage of many eyes reviewing the code, but the developers have less incentive to spend time on anything beyond uninterrupted coding. Consequently, open-source software often suffers from a sore lack of planning, testing and documentation.

I found one example of this when testing Reliance Edge, Datalight’s reliable file system for IoT, on the ARM mbed platform. I teamed with our QA department to write and run a set of agnostic tests on both Reliance Edge and FatFs, the default free file system for mbed. Reliance Edge performed correctly under every test, while FatFs failed some operations in a way that demonstrated the lack of professional testing. <<File system techno language warning>> The most surprising example was in a rewinddir() test (reset a directory read operation), which showed that ARM mbed developers had ignored the FatFs documentation and instead used their intuition to guess how to rewind the directory stream. FatFs documents that f_readdir() should be called with special parameters to rewind a directory stream; but instead, the developers simply modified an index member of the stream. This works fine—until the directory is read all the way through, at which point the index member is ignored. After that, calls to rewinddir() would do absolutely nothing. To be clear, this was not a bug in FatFs, but in the port code that adapts FatFs to ARM mbed. But if the mbed developers had read the FatFs documentation or thoroughly tested their code, they would have discovered the error.

Free software certainly has its place. I’m proud of Datalight for offering Reliance Edge to developers around the world as open source, under the GPLv2 license. But when it comes to professional IoT projects, especially for safety-critical environments, there’s no room to neglect expert developing, testing and documentation. That’s why Datalight also offers a comprehensive test suite and optional MISRA design documentation for commercial Reliance Edge customers. At Datalight, I’ve learned that we can’t settle for carelessly written or under-tested IoT software.

*I don’t say this just to flatter my coworkers. The average Datalight employee in QA, support and engineering has worked at Datalight for 8 years.

Comments (0)

Add a Comment

Allowed tags: <b><i><br>Add a new comment: