Expediting processor verification through testbench infrastructure reuse

by Eric Hennenhoefer and Andrew Betts , TechOnline India - March 03, 2011

Without prejudice to existing verification infrastructure, specialized processor verification IP can free engineers from historical development and maintenance commitments. This liberated time and energy can then allow a renewed focus on verification quality and turnaround times.


You might have thought that processor design was dying out, but think again. The architecture wars rage on between the majors, and plenty of smaller companies still find it worthwhile to develop proprietary architectures (or enhance existing ones) for niche markets. Although fundamental concepts in processor design evolve slowly, supporting technologies are advancing with great enthusiasm.

The growth of these supporting technologies is due, at least in part, to the sheer complexity and dynamics of industrial projects. Powerful algorithms and procedures, dealing with all common obstacles, are tried, tested and generally available. But even still, verification crises such as delayed signoff or even dead silicon are common.

•   61% of new processor designs require a re-spin [IC Insights 2009]
•   48% of total processor development cost are verification related [IC Economics, 2007]
•   55% of all processor designs are delivered late [IC Insights 2009]

This situation means that processor verification is still a major activity in the semiconductor industry, and that reliable and predictable processor verification outcomes remain important, if elusive goals. To reach these goals, the industry must make the inevitable shift toward embracing IP. Without prejudice to existing verification infrastructure, specialized processor verification IP can free engineers from historical development and maintenance commitments. This liberated time and energy can then allow a renewed focus on verification quality and turnaround times.

Testbench infrastructure

Simulation-based processor verification essentially consists of the comparison of a reference model (including an Instruction Set Simulator, or ISS) and the processor RTL using a common set of tests (figure1). The quality of these tests – their coverage of all interesting behavior in the minimum amount of time – is essential to project success. In addition, multiple types of code coverage are performed. Examples of this include RTL code coverage and functional coverage of both the architecture and implementation.


Fig 1: Processor verification testbench

In spite of the structural simplicity of the processor testbench, its complexity often exceeds that of the design under test. The two major contributors to this are the ISS and the random test generators. The complexity of the ISS is a direct reflection of the complexity of the processor architecture. A random test generator, on the other hand, must scale not only with the processor architecture, but also with its implementation.

Achieving maximum infrastructure reuse

Processor projects are not done in isolation, and a major challenge for verification teams is to reuse as much of the infrastructure as possible from one project to the next. The Big Question is, as the verification space grows and changes, how can we achieve the maximum infrastructure reuse?

To address this question, let us examine the idea of the verification space. Figure 2 depicts the verification space as a product of two main features, the complexity of architecture and of implementation.

Fig.2 The verification space

Architectural complexity is a function of the instructions, exceptions and state defined in the Instruction Set Architecture (ISA), including special behaviors and modes. For general-purpose processor architectures, such as ARM or X86, the ISA will be continually expansive over time as the architecture evolves to cover every possible usage while maintaining backwards compatibility.

Implementation complexity grows with the inception of new design features. Even a small increase in implementation complexity can cause huge problems for the verification flow. Examples include adding multiple cores, enhanced pipelines, and memory system optimizations.

When changes to a design are made, the verification space changes in two ways: in general-purpose processor families, where backward code compatibility is an important goal, the project-to-project verification space increases in implementation complexity as new micro-architecture features are added (Y axis of figure 2).

For example, new caching and branch prediction systems might be added to improve performance, or multi-processor operations may have to be supported. This situation is depicted in figure 3, where the new verification space for the test generator to cover is shown in red.




                          Fig.3 Required increment in test generation capability for different project types

In other types of processors, especially DSPs, changes to the core ISA from product to product differ significantly, by the re-encoding of instructions, for example. Even apparently easy changes, such as the addition of a single instruction for optimizing a favorite algorithm, can prove to be very complex for test generation if there are special use cases to take into account. A common source of difficulty is software-exposed VLIW changes, such as new addition slots, interlocks, and so on. These don’t affect the ISA directly, but they certainly affect test generation. This situation is illustrated in figure 3b, where the reusable test collateral is shown to be very small in comparison to the total verification space.

In short, implementation complexity and quickly evolving ISAs can have a huge impact on the ability to reuse testbench infrastructure – and particularly test generators – from project to project.

All too often, the end result is expensive, unscheduled rewrites of infrastructure code.

How project size affects the process

The nature of the project-to-project changes notwithstanding, approaches to processor verification depend on the scale of the undertaking. This may be massive, as in the examples of the Intel Atom and the ARM Cortex, or more moderate. Examples of the latter category include domain-specific, proprietary architectures, and also enhancements to existing ones, such as those of ARC or Tensilica.

Massive projects are attempted only by large companies that have been in the game for a considerable time. Any new processor development therefore leverages existing infrastructure and test suites (backwards compatibility with existing architectures is a frequent requirement), and new tests and testing components are added to this infrastructure as needed.

Such companies generally consider functional verification to be a core competence, and they produce and maintain their own functional verification tools almost entirely in-house. This approach makes sense - up to a point. The production and maintenance of large amounts of code (usually C or C++) costs a lot of money. Further, to get the right return on this investment a company must maintain a strong focus on the software activity. This means well-defined and managed projects for the different components of the functional verification infrastructure, with dedicated and motivated owners. As these infrastructure projects age, this kind of focus can be very hard to maintain. The constraints solver guru leaves the company, gets sick, or is hit by a meteorite, and his replacement wants to "clean up" thousands of lines of code.

Before we consider ways to reduce exposure to such risks, let us turn to the second category of projects. Processor architectures in “moderate” projects are simpler than those of their massive counterparts (at least, they start that way), and the infrastructure is built and maintained by smaller teams. The engineers involved often come from a hardware development background, and this is one of the reasons for the frequent choice of Hardware Verification Languages (HVLs) such as e, Vera and SystemVerilog. These are conceptually very close to Hardware Development Languages (HDLs) such as Verilog and VHDL, which may also be used for verification tasks. An HVL is similar in look and feel to an HDL, but it includes specialized verification features. The most important of these is a constraints engine, which is a difficult function to implement from scratch (though it typically only takes about 1% of the code in a full test generator).

The main drawback of HVL-based verification systems is that they tend to reach a saturation level, beyond which it becomes inefficient and costly to incorporate new features. HVLs may be verification oriented, but they are not processor verification oriented, and using an HVL for processor verification is a little like using Excel to manage company accounts. It’s very useful for getting started, and you need the flexibility it provides in the flow, but once a certain level of complexity is reached you also need to invest in specialized tools. At this point, if you wish to continue along the path of creating and maintaining your own tools, the only choice is to write them at the C/C++ level.

We therefore see that the “massive” and “moderate” project approaches are the same in one crucial respect: they both require a large commitment to coding and maintaining the testbench infrastructure, especially the test generators.

The increasing commitment to infrastructure

Each time a test generator is developed or updated for a new project, the team runs certain risks. An inadequate or inefficient test generator may be unable to sufficiently cover the functionality of the design, or it may be too slow to do so in a reasonable amount of time. It may also lack the needed constraints to test or avoid certain behaviors. If the test generator is late or not completed on schedule, it stands to compromise the schedule for the entire project.

Suppose, for example, that a static test generator (where tests are generated using architecturally visible state only) is discovered to be generating insufficient coverage of a complex processor pipeline. The current coverage metrics are increasing too slowly to meet deadlines, and we realize that our approach is not working. At this point, our options would be to either re-allocate engineering resources to hand-write vast numbers of directed tests or upgrade to dynamic test generation, where corner cases are provoked by using state information that is only available from a running implementation. Of course, the difference between a static and dynamic generator is a fundamental one, and it may be hard to add dynamic capability to an existing, static generator. This illustrates how projects can easily arrive at a verification impasse– new implementation features added for performance, expanding the verification space.

Of course, the ideal situation for a company involved in processor verification is to be able to limit its own work on testbench infrastructure to the strict minimum that differentiates its product. Further, it is preferable to be in the position of evaluating and configuring tools, rather than writing code and scripts, since the latter are more expensive to maintain than configuration tables, for example. This Holy Grail can only be reached if the necessary generic products are available for building the non-differentiating part of the testbench. Fortunately, because processor verification is such a widespread activity, and since modern processor designs generally use a common set of techniques and building blocks, more and more progress is being made in this direction. Fig.4 divides the major functional blocks modern test generators into a test generator core and DUT specific layers. The generator core consists of generic components that may be reused across a wide range of designs. The DUT specific layers are comprised of custom components that are unique to a given design. The 90/10% split is a generalized approximation.


Fig.4: Modern test generator components

Most architectural and implementation features in a new processor are shared by other processors. There are only so many ways to create and test an ADD instruction, or a pipeline, or a multi-threaded architecture. As the architecture * implementation product continues to drive up the complexity of test generators, either teams are going to have to make a much larger commitment to building this infrastructure, or they must start leveraging IP in this field.

Looking forward

In summary, the use of specialized tools in certain key parts of the verification process can and should be utilized in effort to re-asses engineering focus. Rather than being responsible for the evolution and maintenance of major pieces of the testbench, engineers should be directing their efforts on the layer of the verification system that is product-specific. Processor projects suffering from increasing implementation complexity, or from frequent and radical changes in instruction sets, stand to benefit most from this trend. There are particularly attractive opportunities to do this in the processor domain, where similar problems are seen industry-wide and the scope for generic solutions is therefore especially large.


About the authors:
Eric Hennenhoefer, President, CEO & Co-Founder, Obsidian Software Inc. (Austin, Texas), can be reached at eric@obsidiansoft.com.

Andrew Betts is Technical Sales & Marketing Consultant, Iconda Solutions.


blog comments powered by Disqus