Code Coverage Convergence in Configurable IP

by P. Venkata Giri Kumar, Nivin Ninan George, Sarannya K, Sriram Balasubramanian, Srikanth Vadanaparthi , TechOnline India - May 26, 2010

The authors of this paper have defined a method to generate code coverage reports for multiple configurations which reduces additional verification cycles. This methodology has been applied to verify the configurable USB 2.0 HS OTG DesignWare IP Core. The results are provided in the paper which shows significant improvement in the coverage numbers without any additional verification cycles.

Abstract: Code coverage plays an important role in verification. It is essential to meet the target numbers for the code coverage metrics at the end of functional verification. The verification of a configurable IP focuses on verifying the RTL for multiple configurations. The RTL varies with configurations and hence achieving the targeted code coverage for multiple configurations is a verification challenge. The challenge lies in reducing the additional verification cycles to achieve the code coverage goals.

I. Introduction

Code coverage is an essential metric for completion of the verification of a SoC or an IP. The code coverage report helps you identify verification loopholes by analyzing the uncovered coverage metrics in the report. The target coverage numbers are defined for all the code coverage metrics (Line, Toggle, Condition and FSM). Additional verification cycles are required if the code coverage target metrics are not met. It is required to meet 100 percent coverage goals but it does not mean that verification is complete. However, it is an important metric for the measure of the verification. Code coverage metrics must be used in conjunction with other metrics such as functional coverage, assertion coverage, and formal equivalence checkers [1] to measure verification completion.

In general, verification is a two-dimensional problem. The testbench development, coverage convergence (functional and code) are the two dimensions of the verification. The configurable IP provides third dimension to the IP verification namely Multiple Configurations. Verifying the configurable IP across multiple configurations is a challenging task and challenge lies in both the dimensions. Achieving code coverage goals for multiple configurations requires special attention as it might impact the verification schedule if not addressed properly.

II. Multiple Configurations of a Configurable IP

IP designers provide a list of configuration parameters and users can configure the core by selecting the configuration parameters based on their SoC requirements. The selection of a configurable parameter adds a piece of RTL code into the design. Some of the configuration parameters can take multiple values and the RTL code varies according to the value selected for the parameter. The end users can generate multiple configurations by selecting different values for each of the configuration parameters. As a result, the design (RTL code) is different for each configuration.

Table 1 explains this concept using two configuration parameters supported by Designware USB 2.0 HS OTG Core (2).


Table 1. Configuration Parameters
Click on image to enlarge.

The design supports the following architectural modes for the DMA interface:

  • Slave
  • External DMA, and
  • Internal DMA

Similarly it supports the following PHY interfaces:

  • UTMI+
  • ULPI, and
  • UTMI+ and ULPI

Totally, nine configurations can be generated with these two configuration parameters and the RTL code is going to vary across these configurations. There are 39 configuration parameters in the Designware USB 2.0 HS OTG Core and each of them takes multiple values. As the number of configuration parameters increase, the number of possible configurations increases. A total of sixty configurations generated out of 39 configuration parameters are considered for regressions. The code coverage is a measure of the RTL code covered in the verification of a selected configuration and hence the code coverage needs to be measured for each of the configurations.

{pagebreak}III. Coverage Convergence

Code coverage is an important metric for the closure of verification and is used to measure the quality of verification and in turn quality of the core. As explained in Section II, the RTL varies across configurations and therefore the code coverage needs to be generated for every configuration. This is a typical verification problem seen in configurable IPs. Additional verification cycles are often required to achieve code coverage goals for multiple configurations. This section discusses some existing code coverage generation methods along with their limitations. This section also proposes a new method for generating code coverage reports for multiple configurations.

A. Method 1:

  • 1. Identify the configuration with the maximum RTL overlap with all other configurations. This configuration is sometimes referred to as a “golden” configuration.
  • 2. Run regressions on this configuration with code coverage enabled.
  • 3. Analyze the code coverage results and fine tune the tests to meet the target numbers.

These code coverage numbers are quoted to be the final numbers applicable to all the configurations. The reason for such an approach is that the code coverage numbers reflect the status of the maximum code across all the configurations. However, the limitation with such an approach is that the code coverage numbers are accurate for the selected configuration only, and do not give the exact coverage status for the remaining configurations.

B. Method 2:

Run regressions for all configurations with coverage enabled. The successful completion of regressions ensures that the functional coverage and code coverage goals are met. This approach ensures that the code coverage numbers are accurate for every configuration. This approach seems to be good as we achieve 100% coverage goals for all configurations. However, as number of configurations increase, additional verification cycles are required to achieve code coverage goals thereby impacting project schedules.

C. Method 3 - The Proposed Method:

This section describes a method to generate code coverage for multiple configurations. This method overcomes the drawbacks of the above methods.


Figure 1: USB Testbench Block Diagram
Click on image to enlarge.

A Constrained Random Verification (CRV) environment as shown in Figure 1 is used for the verification of Designware USB 2.0 HS OTG Core. The DUT (HS OTG) has an AHB interface at one end and a USB interface at the other end. The application drivers are developed based on AMBA VIP and USB drivers are developed based on USB VIP. The environment is capable of generating random and directed stimulus, which will be used by these drivers to generate meaningful transactions.

The scoreboard collects the transactions from the AHB and USB interfaces and verifies the correctness of the DUT. The Functional Coverage model and Assertions are developed to measure the functional coverage of the verification. The “always ON” and power isolation cells are implemented to support hibernation and low power features. The environment addresses the requirements of multiple DUT configurations and is used to verify all the supported DUT configurations.

The verification focuses on running regressions for all the supported configurations and coverage is enabled for all the regressions. The verification is complete once the following goals are met for all configurations:

  • 1. All the defined tests to pass
  • 2. Functional Coverage goals to be met
  • 3. Code Coverage goals to be met for all configurations

At the end of the regressions a consolidated report across all configurations is generated for all the defined goals. It is relatively easy to generate the consolidated report for functional coverage as the tool merges the data effectively across configurations. However, you cannot generate a consolidated code coverage report as the RTL code varies across the configurations. You must generate code coverage reports for all configurations independently. As a result, additional verification cycles may be required to meet the code coverage goals even after the first two goals are met (all the tests have passed, and a 100 percent functional coverage goal is achieved). The proposed method either reduces or eliminates these additional verification cycles.

The Unified Report Generator [3] (URG) is a feature provided by VCS that is used to generate code coverage reports. The code coverage report is generated using the urg command as shown below:

urg -dir Config1/simv.cm … Config60/simv.cm

where:

  • Config1 to Config60 are the configurations that are generated as defined in Section II, and,
  • simv.cm is the coverage database generated by VCS on enabling the code coverage.
This method introduces the following concepts:
  • Base design and sub-design
  • Common RTL and distinct RTL

The following sections explain these concepts in more detail.

{pagebreak}1) Base design and Sub-design:

The first design encountered by the coverage tool is defined as base design and the remaining designs are defined as sub-designs. In the above example for urg, Config1 is the base design and Config2 to Config60 are the sub designs.

2) Common RTL and Distinct RTL:

Common RTL is the code in the sub-design that is common to the base design (Config1) and the sub-designs (Config2 to Config60).

Distinct RTL is the code in the sub-design that is not part of the base design. The Distinct RTL code is included with the selection of the configuration parameters as explained in Section II.

The code coverage report is generated by specifying one of the configurations as the base design and remaining designs as sub-designs. The coverage tool merges the coverage data of the sub-designs with the coverage data of the base design.

The line coverage is used as an example to explain the process of merging line coverage data from the sub-designs to the base design. Table 2 lists the scenarios encountered by the coverage tool while merging the line coverage data across the different configurations and the appropriate actions taken.


Table 2: Scenarios encountered by the Code Coverage Tool and the Actions Taken
Click on image to enlarge.

The coverage reports for other coverage metrics like toggle, condition and fsm can be generated based on the same process.

The final merged report shows the code coverage for the base design (Config 1 in this case) and it includes the coverage data for the base design and the coverage data for the Common RTL from the sub-designs. If code coverage is generated by considering only the RTL for Config1 (base design) then the coverage data for the common RTL in the sub-design is not considered and therefore the coverage numbers might be less when compared to the merged report.

The generation of merged coverage report is repeated multiple times to generate the code coverage for multiple configurations and each time the base design points to a different configuration and the remaining configurations become the sub-designs as shown in Table 3. The table shows the merged coverage report for the four configurations by merging the base design with the remaining 59 sub-designs. The final coverage report in the table reflects the coverage status for the respective base designs.


Table 3. Base and Sub-design Distributions in Merging the Coverage
Click on image to enlarge.

In Method 1, the max configuration is used to generate the coverage numbers and the data is accurate for the max configuration but it may not be accurate for other configurations. Whereas using this method the coverage report is generated for every configuration and hence the data will be accurate for every configuration.

In Method 2, additional verification cycles are required to improve coverage numbers. Method 3 enables you to generate coverage numbers by merging the coverage data in the sub-designs with the base design without any additional verification cycles.

In the above example a set of 60 configurations are taken to explain the code coverage convergence. However, as the number of configurations increase, the RTL overlap across the different configurations is greater and therefore the coverage numbers for all the configurations improve.

D. Results

The regressions were run for 60 configurations and the code coverage reports were generated using the VCS urg command. The code coverage reports were generated for every configuration without merging the coverage data across configurations. Table 4 shows the results for the four configurations.


Table 4. Individual Coverage Reports for 4 Configurations
Click on image to enlarge.

{pagebreak}Table 5 shows the merged coverage report for the four configurations. This report was generated using Method 3.


Table 5. Merged Coverage Report for 4 Configurations
Click on image to enlarge.

The merged coverage report shows significant improvement in the coverage numbers when compared to individual coverage reports. No additional verification cycles are involved in improving the code coverage numbers. The merged coverage reports of all the configurations were analyzed and additional tests were added to meet the code coverage goal. The advantage with this method is that it is simpler to generate the coverage report for customer-specific configurations by specifying that configuration as the base design and the remaining configurations as sub-designs.

IV. Conclusions

This article proposes an elegant method to generate the code coverage reports for multiple configurations. The method has been applied to the verification of our configurable Designware Core USB 2.0 HS OTG.

V. References:

  1. Coverage is the heart of verification
  2. Designware Core USB 2.0 HSOTG IP
  3. Synopsys Unified Coverage Reporting User Guide
  4. Writing Testbenches, Janick Bergeron, Springer US
  5. Coverage-Driven Verification, Andrew Piziali, Springer US

About the Authors :

The authors are with Synopsys (India) Private Limited. Kumar is Senior R & D engineer, IP Engineering, and can be reached at vgk@synopsys.com; Balasubramanian is R & D manager, IP Engineering, and can be reached at bsriram@synopsys.com, while George, reachable at nivin@synopsys.com; Sarannya, reachable at sarannya@synopsys.com, and Srikanth, who can be reached at srika@synopsys.com, are R & D engineers, IP Engineering.

Comments

blog comments powered by Disqus