Standards Certification Testing: Bringing Order to the Internet of Things

This is a sponsored article brought to you by Comarch.

From human speech to quantum encryption, communication relies on protocols that are mutually understood and reliably implemented. Without them, there is only noise.

In the world of electronic communications, standards certification makes sure that everybody knows the rules and plays by them. Comarch-the $500-million IT company with headquarters in Krakow, Poland, and offices in 100 countries-is playing a leading role in developing automated standards certification systems.

In 2007, Comarch began working with the UPnP Forum (now part of the Open Connectivity Foundation) to certify Universal Plug and Play compliance for personal computers, Wi-Fi access points, routers, audiovisual, and other devices on home-scale networks.

Over the years, Comarch has steadily expanded its work with multiple standards organizations.

In October 2021, when the AVNU Alliance debuted its test tool for the Milan Advanced Certification Program (MACP) for time-sensitive networks (TSNs), Comarch distilled 15 years of automated certification testing into its new Comarch Automated Testing Framework (CATF) to make it happen.

Thanks to CATF, AVNU could promise its members faster, easier, more convenient, and less expensive testing" that would guarantee performance and interoperability of high-speed devices for audiovisual, automotive, and industrial applications-applications that demand security, reliability, efficient use of available bandwidth, adaptation to varying latency, and ultra-precise timing.

CATF Product Demo

CATF (Comarch Automated Test Framework) is a new product in Comarch's portfolio. It's a specialist conformance test framework dedicated for certification alliances, designed to shorten development time and ensure high stability and maturity.

The Business Case

Standards organizations and their members are looking to faster, digital, cloud-based testing models to help them respond to the increasing pace of technology development. More and more, that requires an expert partner.

Standards organizations know their own technologies inside-out. They can define the requirements very precisely. They are able to identify needs and goals," says Radoslaw Kotewicz, Comarch's director of IoT Sales and Consulting. But he explains that in most cases-not all, but most-they lack in-house expertise in certification test development.

In the past, says Kotewicz, standards alliances nonetheless developed their own certification tools-usually using the resources of their member organizations. That could be a drawn-out process, but it wasn't major concern...then.

Even five or ten years ago," says Kotewicz, developing a typical product might take up to two years." In that context, it wasn't a problem if it took six to twelve months to develop automated standards-certification tools.

Now, though, manufacturers are trying to roll out new products faster," he says. "Standards organizations are accelerating their development schedules to adapt, and may need to deliver new automated testing tools in as little as three months."

Over the years, more and more standards organizations have moved from dedicated in-house tools to external tools. They know their own technologies, but they are finding that organizations like Comarch, which specialize in standards testing, allow them to bring their automated tests online much faster."

To accelerate standards-certification test development, speed up device certifications, and improve interoperability testing, the company built the Comarch Automated Test Framework. Written in .Net C# running under Windows, CATF uses libraries of Python test scripts. Most communications protocols use standard types of input and produce standard types of output that are readily interpreted.

Download Free Whitepaper on CATF

Download Comarch's free whitepaper "Technology Certification and Standardization Process with Comarch Automated Test Framework" (CATF) to learn more about the company's testing and certification tools.

Certain protocols, however, require nonstandard calls and responses, while others may require more detailed monitoring at the hardware level-via either a standard piece of test equipment like a commercial vector network analyzer or, infrequently, a piece of custom hardware designed and built by Comarch.

Modularity gives CATF its flexibility-flexibility for proprietary tests, for accommodating standards that are still in development, for looping in uncommon hardware, or for validating interoperability with a wide range of other devices and protocols.

The power of the solution," says Kotewicz, is the common way to manage the text cases, letting us add or extend test cases (to accommodate evolving interoperability requirements, for example)."

Long Experience

The company has long experience working with standards organizations, their member companies, their vendors and contractors, and authorized testing laboratories (ATLs) to develop, roll out, and manage certification tests. To consider just a few of Comarch's engagements:

UPnP and OCF

When the Open Connectivity Foundation (OCF) took over management of UpnP in 2016, UpnP joined a stable of standards for the wider Internet of Things. Comarch came along, too, winning OCF Outstanding Contributor Awards in 2017 and 2020. As OCF commented:

As the developer of the OCF Certification Test Tool [CTT], Comarch has demonstrated strong commitment to establishing OCF as a leading communication standard. Comarch has hosted and organized...face-to-face meetings and worked with specification authors and work groups to resolve issues and refine OCF requirements, as well as cooperating with IoTivity [open-source IoT software] developers to improve security requirements in the CTT and in specifications. Comarch's proactive and responsive approach to resolving issues has led to meeting tight deadlines and making consistent improvements to the OCF CTT."

AirFuel

Comarch has a history of solving demanding problems. Since 2014, it has worked with the AirFuel Alliance (airfuel.org) to create an automated test system that could tackle the combined software and hardware challenges of resonant wireless power transfer-and certify that AirFuel compliant devices will operate safely and reliably in the increasingly complex interconnected world.

The application was demanding: in addition to standard communications I/O, it had to monitor and validate some hardware conditions (such as the flow of power through the PRU power receive unit) and interact with a variety of external devices. The certification tests were complex, and AirFuel's first certification rounds required up to two costly weeks. Experience and CATF automation reduced that time to less than a day.

FiRa

Since 2020, Comarch has worked with the FiRa Consortium (firaconsortium.org). FiRa's mission is to foster a robust [ultra-wide-band] ecosystem to enable rapid technology deployment," for a host of smart city and smart home IoT applications like transportation ticket validation, indoor navigation, asset tracking, social-distancing and contact tracing, unmanned store access, residential access control. Comarch used CATF to build FiRa's MAC Conformance Test Tool.

Lessons Learned

Along with the Comarch Automated Test Framework and a variety of test cases ready to use or adapt, Comarch brings this customer experience-and a lot more-to building reliable, cost-effective certification tests in record time.

In the process, Comarch has learned some important lessons:

• Implement the most efficient possible test process. Time saved in a single round is multiplied by the number of devices tested, and the total savings can be substantial.
• Start developing the certification test process as early as possible. But when that means starting before the standard is final, be sure you can add, subtract, or modify test cases on the fly.
• Give the standards organizations with the tools they need to understand the test-building process. Having a partner who knows what questions to ask make's the process much easier.
• Be prepared for interesting problems. In one case, Kotewicz says, the standard required power to ramp up so quickly that only one or two devices in the world could measure the change. In another case, the language of the standard was initially so tangled that it was impossible to verify as written.
• And, in the Internet of Things-or, indeed, any environment in which multiple standards are in operation at the same time-a modular, multi-protocol approach like Comarch's is invaluable.

The process is demanding, complex, and we know it by heart," says Kotewicz.