Verification Horizons BLOG

This blog will provide an online forum to provide weekly updates on concepts, values, standards, methodologies and examples to assist with the understanding of what advanced functional verification technologies can do and how to most effectively apply them. We're looking forward to your comments and suggestions on the posts to make this a useful tool.

19 February, 2015

It’s amazing how quickly a year goes by. DVCon 2014 seems like it was just a few months ago, and here we are rolling up on DVCon 2015. As my colleague Dennis Brophy blogged earlier, it was just a year ago that Mentor Graphics proposed that Accellera launch a Proposed Working Group (PWG) to explore whether sufficient need and interest existed in the industry to standardize a portable stimulus specification. After nearly a year of requirements gathering and discussion between the participants of the Portable Stimulus PWG, the PWG announced that it had concluded there was sufficient interest and need to justify an official standards body.

When portable stimulus is discussed, it is often in the context of SoC-level verication – both because of how critical SoC- and System-level verification is today, and because of the stresses an SoC-level environment places on a portable stimulus solution. SoC-level verification must be done in simulation, emulation, and silicon, and is typically driven via the embedded processors in the design – in other words, using the design to verify itself. In an SoC-level environment, Portable Stimulus introduces a degree of stimulus-generation automation not possible using directed tests, while enabling the generated stimulus to be portable across engines and tailored to the performance characteristics of those engines.

While SoC-level verification may illustrate an extreme example of the requirements for portable stimulus, there are many other cases where portable stimulus is extremely valuable. One of these cases is illustrated by a paper co-authored by Boris Hristov from Ciena , and my colleague, Mike Andrews. Portable Stimulus Models for C/SystemC, UVM and Emulation discusses how portable stimulus can be applied to verify a C/SystemC design that will be targeted at High-Level Synthesis (HLS). Then, reuse the same stimulus model to verify the RTL output of the High-Level Synthesis tool in a UVM environment in simulation and emulation. While a very different target than SoC-level verification, the benefits of applying Portable Stimulus are much the same – consistent stimulus across languages and engines, and a high degree of automation.

So, if you’re attending DVCon be sure to check out the presentation for Portable Stimulus Models for C/SystemC, UVM and Emulation. It’s on Tuesday afternoon in the Multi-Language session: http://dvcon.org/content/event-details?id=180-7

Come find me in the Mentor Graphics booth (booth 801), and I’ll be happy to discuss verification in general and Portable Stimulus, specifically, in more depth with you.


11 February, 2015

Accellera Approves Creation of Portable Stimulus Working Group

At DVCon 2014, Mentor Graphics proposed Accellera launch an exploratory exercise, called a Proposed Working Group (PWG), to determine if there was sufficient interest and need to create a standard in this area.  To help motivate the consideration of this activity, we indicated we would offer our graph-based test specification embodied in our inFact verification tool.

Rapid adoption of our technology has been the trend, especially when used in conjunction within a SystemVerilog UVM testbench environment.  One of the major benefits of UVM has been the portable nature of the testbench to facilitate design verification within and across companies.  The exclusive nature of our graph-based test specification language limits its easy use within the industry leading users to suggest we look to standardize it in keeping with the fundamental UVM principle of testbench portability.

After about a year of discussion in Accellera, the group announced it had concluded there should be an official standards project in this area.  Industry participants have likewise offered quotes of support for the formation of the Accellera Portable Stimulus Working Group.

The challenges to efficient and effective verification continue to grow.  If we stop where we are today in verification algorithm advances and standards the trend to require more people, time or compute resources will continue grow unabated at exponential rates.

For Mentor Graphics part, the verification team here has gone to market with innovative technology that has shown remarkable ability to improve verification productivity and efficiency.  The specification we offer to Accellera to seed this project is the same embodied in technology we used when we partnered with TSMC to validate advanced functional verification technology we announced in 2011.  From that announcement, we shared that tests conducted by AppliedMicro in designs destined for TSMC shortened “time-to-coverage by over 100x.”

One need not wonder if it is possible to shrink a month’s worth of verification tests into less than an 8 hour work day.  It is.  To find out how our specific use of this technology works and what motivates us to support standardization of Portable Stimulus in Accellera, I invite you to visit the Verification Academy where a session on Intelligent Testbench Automation shows what can be done.

And for those who would like to help in the development of the standard and may have technology to further underpin it, you should consider attending the first organizational meeting of the Portable Stimulus Working Group at DVCon 2015 March 5th from 6pm-9pm.  Contact Accellera for member-only meeting details or catch me at DVCon 2015 and I can share more information with you.

, , , , , , ,

8 February, 2015

FPGA Design Trends

In my previous blog, I introduced the 2014 Wilson Research Group Functional Verification Study (click here). The objective of my previous blog was to provide an overview on our large, worldwide industry study. The key findings from this study will be presented in a set of upcoming blogs. In this blog, I present trends related to various aspects of FPGA design to illustrate growing design complexity.

Let’s begin by examining embedded processor trends targeted at a general FPGA implementation. Our 2014 study found that 56% of all FPGA designs contained one or more embedded processors, as shown in Figure 1. Although we did not see an overall growth in the number of FPGAs containing one or more embedded processors between 2012 and 2014, we did see an increase in the number of FPGA projects creating designs containing more than one embedded processor.

2014-WRG-BLOG-FPGA-1-1

Figure 1. Number of embedded processors in FPGA trends

SoC class designs (i.e., designs containing embedded processors) add a new layer of verification complexity to the verification process that did not exist with traditional non-SoC class designs due to hardware and software interactions, new coherency architectures, and the emergence of complex network-on-a-chip interconnect.

In addition to embedded processors targeted at general FPGA class of designs, there has been a recent emergence of specific programmable SoC FPGA implementations, such as: Xilinx’s Zynq, Altera’s Arria/Cydone, and Microsemi’s SmarFusion. Figure 2 shows the adoption trends for these programmable SoC FPGAs, which you can see grew by over 93 percent between 2012 and 2014. Keep in mind that this trend data does not represent volume production—it represents the number of FPGA projects that are creating designs targeted at a programmable SoC class of FPGA.

2014-WRG-BLOG-FPGA-1-3

Figure 2. Type of FPGA implementation trends

As the industry moves to SoC class designs, regardless of targeted FPGA implementation, FPGA projects are starting to increase their adoption of industry standard on-chip bus protocols—versus proprietary bus protocols. Figure 3 shows the current adoption of AMBA and other on-chip bus protocols for FPGA designs as identified by our new study. Note, the reason we are not showing trends here is that the 2012 study did not separate out the various AMBA protocols, which is something we decided to do for our 2014 study. Hence, we cannot do an apples-to-apples comparison between 2012 and 2014 for FPGA on-chip bus protocol adoption.

2014-WRG-BLOG-FPGA-1-3a

Figure 3. FPGA on-chip bus protocol adoption

Another aspect of SoC class design is the emergence of IP-based design practices, which is fundamental for improving design productivity. Figure 4 shows FPGA design composition trends—and we see that there has been a declined in new logic created by FPGA project teams. At the same time we see an increase in the adoption of both internally developed and externally acquired IP.

2014-WRG-BLOG-FPGA-1-5

Figure 4. FPGA design composition trends

In my next blog (click here), I’ll focus on verification effort trends related to FPGA designs.

Quick links to the 2014 Wilson Research Group Study results

,

21 January, 2015

This blog is a continuation of a series of blogs that present the highlights from the 2014 Wilson Research Group Functional Verification Study (for a background on the study, click here).

In this blog I discuss the issue of study bias, and what we did to address these concerns.

MINIMIZING STUDY BIAS

When architecting a study, three main concerns must be addressed to ensure valid results: sample validity bias, non-response bias, and stakeholder bias. Each of these concerns is discussed in the following sections, as well as the steps we took to minimize these bias concerns.

Sample Validity Bias

To ensure that a study is unbiased, it’s critical that every member of a studied population have an equal chance of participating. An example of a biased study would be when a technical conference surveys its participants. The data might raise some interesting questions, but unfortunately, it does not represent members of the population that were unable to participant in the conference. The same bias can occur if a journal or online publication limits its surveys to only its subscribers.

A classic example of sample validity bias is the famous Literary Digest poll in the 1936 United States presidential election, where the magazine surveyed over two million people. This was a huge study for this period in time. The sampling frame of the study was chosen from the magazine’s subscriber list, phone books, and car registrations. However, the problem with this approach was that the study did not represent the actual voter population since it was a luxury to have a subscription to a magazine, or a phone, or a car during The Great Depression. As a result of this biased sample, the poll inaccurately predicted that Republican Alf Landon versus the Democrat Franklin Roosevelt would win the 1936 presidential election.

For our study, we carefully chose a broad set of independent lists that, when combined, represented all regions of the world and all electronic design market segments. We reviewed the participant results in terms of market segments to ensure no segment or region representation was inadvertently excluded or under-represented.

Non-Response Bias

Non-response bias in a study occurs when a randomly sampled individual cannot be contacted or refuses to participate in a survey. For example, spam and unsolicited mail filters remove an individual from the possibility of receiving an invitation to participate in a study, which can bias results. It is important to validate sufficient responses occurred across all lists that make up the sample frame. Hence, we reviewed the final results to ensure that no single list of respondents that made up the sample frame dominated the final results.

Another potential non-response bias is due to lack of language translation, which we learned during our 2012 study. The 2012 study generally had good representation from all regions of the world, with the exception of an initially very poor level of participation from Japan. To solve this problem, we took two actions:

  1. We translated both the invitation and the survey into Japanese.
  2. We acquired additional engineering lists directly from Japan to augment our existing survey invitation list.

This resulted in a balanced representation from Japan. Based on that experience, we took the same approach to solve the language problem for the 2014 study.

Stakeholder Bias

Stakeholder bias occurs when someone who has a vested interest in survey results can complete an online study survey multiple times and urge others to complete the survey in order to influence the results. To address this problem, a special code was generated for each study participation invitation that was sent out. The code could only be used once to fill out the survey questions, preventing someone from taking the study multiple times or sharing the invitation with someone else.

2010 Study Bias

While architecting the 2012 study, we did discover a non-response bias associated with the 2010 study. Although multiple lists across multiple market segments and across multiple regions of the world were used during the 2010 study, we discovered that a single list dominated the responses, which consisted of participants who worked on more advanced projects and whose functional verification processes tend to be mature. Hence, for this series of blogs we have decided not to publish any of the 2010 results as part of verification technology adoption trend analysis.

The 2007, 2012, and 2014 studies were well balance and did not exhibit the non-response bias previously described for the 2010 data. Hence, we have confidence in talking about general industry trends presented in this series of blogs.

Quick links to the 2014 Wilson Research Group Study results

,

21 January, 2015

This is the first in a series of blogs that presents the findings from our new 2014 Wilson Research Group Functional Verification Study. However, unlike my previous Wilson Research Group functional verification study blogs, which focused on the ASIC/IC market, I plan to begin this set of blogs with an exclusive focus on FPGA trends. Why? For the following reasons:

  1. Unlike the traditional ASIC/IC market, there has historically been very few studies published on FPGA functional verification trends. We started studying the FPGA market segment back in the 2010 study, and we now have collected sufficient data to confidently present industry trends related to this market segment.
  2. Today’s FPGA designs have grown in complexity—and many now resemble complete systems. The task of verifying SoC-class designs is daunting, which has forced many FPGA projects to mature their verification process due to rising complexity. The FPGA-focused data I present in this set of blogs will support this claim.

My plan is to release the ASIC/IC functional verification trends through a set of blogs after I finish presenting the FPGA trends.

Introduction

In 2002 and 2004, Collett International Research, Inc. conducted its well-known ASIC/IC functional verification studies, which provided invaluable insight into the state of the electronic industry and its trends in design and verification at that point in time. However, after the 2004 study, no additional Collett studies were conducted, which left a void in identifying industry trends. To address this dearth of knowledge, four studies were commissioned by Mentor Graphics in 2007, 2010, 2012, and 2014, which focused on functional verification. These were world-wide, double-blind, functional verification studies, covering all electronic industry market segments. To our knowledge, the 2014 study was the largest functional verification study ever conducted. This set of blogs presents the findings from our 2014 study and provides invaluable insight into the state of the electronic industry today in terms of both design and verification trends.

Study Background

Our study was modeled after the original 2002 and 2004 Collett International Research, Inc. studies. In other words, we endeavored to preserve the original wording of the Collett questions whenever possible to facilitate trend analysis. To ensure anonymity, we commissioned Wilson Research Group to execute our study. The purpose of preserving anonymity was to prevent biasing the participants’ responses. Furthermore, to ensure that our study would be executed as a double-blind study, the compilation and analysis of the results did not take into account the identity of the participants.

For the purpose of our study we used a multiple sampling frame approach that was constructed from eight independent lists that we acquired. This enabled us to cover all regions of the world—as well as cover all relevant electronic industry market segments. It is important to note that we decided not to include our own account team’s customer list in the sampling frame. This was done in a deliberate attempt to prevent biasing the final results. My next blog in this series will discuss other potential bias concerns when conducting a large industry study and describe what we did to address these concerns.

After data cleaning the results to remove inconsistent or random responses (e.g., someone who only answered “a” on all questions), the final sample size consisted of 1886 eligible participants (i.e., n=1886). To put this figure in perspective, the 2004 Collett study sample size consisted of 201 eligible participants.

Unlike the 2002 and 2004 Collett IC/ASIC functional verification studies, which focused only on the ASIC/IC market segment, our studies were expanded in 2010 to include the FPGA market segment. We have partitioned the analysis of these two different market segments separately, to provide a clear focus on each. One other difference between our studies and the Collett studies is that our study covered all regions of the world, while the original Collett studies were conducted only in North America (US and Canada). We have the ability to compile the results both globally and regionally, but for the purpose of this set of blogs I am presenting only the globally compiled results.

Confidence Interval

All surveys are subject to sampling errors. To quantify this error in probabilistic terms, we calculate a confidence interval. For example, we determined the overall margin of error for our study to be ±2.19% at a 95% confidence interval. In other words, this confidence interval tells us that if we were to take repeated samples of size n=1886 from a population, 95% of the samples would fall inside our margin of error ±2.19%, and only 5% of the samples would fall outside.

Study Participants

This section provides background on the makeup of the study.

Figure 1 shows the percentage of overall study participants by market segment.

2014-WRG-BLOG-P-1

Figure 1: Study participants by market segment

Figure 2 shows the percentage of overall study eligible participants by their job description. An example of eligible participant would be a self-identified design or verification engineer, or engineering manager, who is actively working within the electronics industry. Overall, design and verification engineers accounted for 60 percent of the study participants.

2014-WRG-BLOG-P-2

Figure 2: Study participants job title description

Before I start presenting the findings from our 2014 functional verification study, I plan to discuss in my next blog (click here) general bias concerns associated with all survey-based studies—and what we did to minimize these concerns.

Quick links to the 2014 Wilson Research Group Study results

,

14 January, 2015

“Who Knew?” about verification IP (VIP), was the theme of a recent DeepChip post by John Cooley on December 18.  More specifically the article states, “Who knew VIP was big and that Wally had a good piece of it?”  We knew.

We knew that ASIC and FPGA design engineers can choose to buy design IP from several alternative sources or build their own, but that does not help with the problem of verification.  We knew that you don’t really want to rely on the same source that designed your IP, to test it.  We knew that you don’t want to write and maintain bus functional models (BFMs) or more complete VIP for standard protocols.  Not that you couldn’t, but why would you if you don’t have to?

We also knew that verification teams want easy-to-use VIP that is built on a standard foundation of SystemVerilog, compliant with a protocol’s specification, and is easily configurable to your implementation.  That way it integrates into your verification environment just as easily as if you had built it yourself.

Leading design IP providers such as ARM®, PLDA, and Northwest Logic knew that Mentor Graphics’ VIP is built on standards, is protocol compliant, and is easy to use.  In fact you can read more about what Jim Wallace, systems and software group director at ARM; Stephane Hauradou, CTO of PLDA; and Brian Daellenbach, president of Northwest Logic; have to say about Mentor Graphics’ recently introduced EZ-VIP technology for PCIe 4.0 (at this website http://www.mentor.com/company/news/mentor-verification-ip-pcie-4 ), and why they know that their customers can rely on it as well.

Verification engineers knew, too.  You can read comments from many of them (at Cooley’s website http://www.deepchip.com/items/dac14-06.html ), about their opinions on VIP.  In addition, Mercury Systems also knew.  “Mentor Graphics PCIe VIP is fully compliant with the PCIe protocol specification and with UVM coding guidelines. We found that we could drop it into our existing environment and get it up and running very quickly”, said Nick Solimini, Consulting DV Engineer at Mercury Systems. “Mentor’s support for their VIP is excellent. All our technical questions were answered promptly so we were able to be productive throughout the project”.

So, now you know,  Mentor Graphics’ Questa VIP is built on standard SV UVM, is specification compliant, is easy to get up and running and is an integral part of many successful verification environments today.  If you’d like to learn more about Questa VIP and Mentor Graphics’ EZ-VIP technology, send me an email, and I’ll let you in on what (thanks to Cooley and our customers) is no longer the best kept secret in verification.  Who knew?

, , , , , , , ,

6 January, 2015

2014 was an exciting year for formal verification to say the least, and below I call out a sampling of noteworthy conference papers that contributed to this energy. Specifically, they show how formal property checking itself is simultaneously scaling to tackle ever larger verification problems, and also being extended to cover domains that were once exclusively the province of simulation.

———-
DVCon Oski Samsung image

Conference: DVCon 2014

Title: Sign-off With Bounded Formal Verification Proofs

Highlights: This paper totally busts the myth that you need to fully solve a formal proof in order to be valuable for verification.  Instead, the authors’ show how “bounded proofs” – i.e. formal proofs that haven’t reached a solution yet – can provide engineers enough information to confidently make sign-off decisions.  Specifically, they show that as long as a given formal analysis isn’t failing as it proceeds deeper in to the state space, and as long as it touches the cover points you wanted to hit, “no news is good news” and the design is likely sound.

Authors: NamDo Kim and Junhyuk Park of Samsung Electronics Co., Ltd.; HarGovind Singh and Vigyan Singhal of Oski Technology, Inc.

Proceedings: http://events.dvcon.org/events/proceedings.aspx?id=163-2

———-
U2U Oracle April

Conference: Mentor Graphics’ User2User Conference, San Jose, CA, April 2014

Title: Targeted Approach to Formal Model Checking:  A Case Study

Highlights: This paper documents a real work SoC verification case study on how formal was focused on three major verification flows: bug hunting, “assurance” (a/k/a proofs of key functions), and exhaustive static & temporal connectivity checking.  The results were impressive: in 50% of the time vs. the simulation approaches they supplanted,  these formal applications found 4 RTL bugs missed by other methods, and 6 more RTL bugs in cache verification – 1 of which would have been a chip killer had it escaped.

Author: Ram Narayan, Oracle Labs

Proceedings: http://supportnet.mentor.com/member/u2u/2014-san-jose.cfm

———-
U2U Qualcomm October image

Conference: Mentor Graphics’ User2User Conference, Bangalore India, October 2014

Title: Gear up ! Speed up ! –   Dead Code Detection Through Reachability Analysis (D2CTRA) for Coverage Convergence

Highlights: For D&V engineers, few things in verification are more frustrating than watching the code coverage score plateau well short of your coverage spec. No matter how many clever directed test you write, no matter how many different random seeds you try, the coverage score simply flat-lines. In parallel, specifying waivers to deliberately exclude unused IP configurations from the analysis can be a tedious and error prone manual process.  In this presentation the authors show how they used Questa CoverCheck to solve this problem on a very control intensive design.  In a nutshell, they were able to rapidly identify dead code areas, easily set coverage waivers on properly dead areas to keep them from inaccurately pulling down the coverage score, and investigate and fix the trouble spots.  (Note: this presentation won the “Best Paper” award for the conference’s functional verification track!)

Authors: Pandithurai Sangaiyah, Anand Kumar of Qualcomm India Pvt Ltd

Proceedings: https://verificationacademy.com/resource/42639

———-

Again, the above is just a sampling of the great work and innovations being made in the formal and assertion-based verification domain in 2014.  If the anecdotes and success stories that I’m regularly receiving in my InBox (let alone the upcoming conference papers and posters that I’m aware of) are any indication, 2015 promises to be an even bigger year for formal-based solutions!

Joe Hupcey III

P.S. Bonus: I’ve already written about the ARM Techcon 2014 paper on Advanced Verification Management and Coverage Closure Techniques by Nguyen Le, Microsoft, et.al.; but it bears re-referencing my prior Verification Horizon’s post on this paper: ARM® Techcon Paper Report: How Microsoft Saved 4 Man-Months Meeting Their Coverage Closure Goals Using Automated Verification Management & Formal Apps

12 December, 2014

Just in time for Christmas and other year-end holidays, I am pleased to announce that the latest issue of Verification Horizons is now available on-line. As usual, we have a great lineup of articles that I’m sure you’ll find informative:

By the way, if you’d like to subscribe to our quarterly Verification Horizons newsletter, you may do so here.

Whichever holiday(s) you celebrate at this special time of year, I hope you experience the peace, joy and hope that my family and I will share this Christmas.

24 November, 2014

SystemVerilog Testbench Debug – Are we having fun yet?

Fun

Debug should be fun. Watching waveforms march by, seeing ERRORS and WARNINGS pop out in a transcript file, tracing drivers back to their source, understanding race conditions between simulators and between source code changes – and my favorite – debugging random stability issues. Fun.

Old School – logfiles and interactive

Or at least it should be fun. It used to be fun. I’d setup my collection of scripts to run tests and examine logfiles. Push the button and go for coffee or go home. The next day I’d examine log files and figure out what happened. Usually I’d have to jump into interactive simulation and debug on the fly. Set some breakpoints and watch what happened. That was then. My tests and RTL were all Verilog. Life was good. I was in control of what was going on, and could get my head around it.

New School – logfiles, interactive and class handles

Fast-forward to today. Still have scripts to run tests. Still have log files. Still push the button and get coffee or go home. Still jump into interactive simulation. Still set breakpoints. But now my tests are SystemVerilog class-based – usually UVM. My tests are C code. My tests are constrained random tests. Debug just got harder. I can’t fit the whole testbench + RTL into my head at once. I need help.

Debugging your class based testbench

I prefer to do as much debug as possible in “post-sim” mode. I want to run simulation and capture as much as possible. Then debug my wavefile and source code. What to do about my SystemVerilog class based testbench? Easy. Capture my classes in the wave database. Show them to me in the wave window.

<UVM Testbench class hierarchy window and those same classes in the wave window>

Wave Window

Wave Window

But that’s not possible. Is it? What IS possible?

What? Objects in the wave database? Yes. Objects and their members in the wave database.

Examine the values of class member variables in post-sim mode. Use the waveform window for classes and class member variables just like signals.

What about the handles that are in my classes? Can I chase them to other objects? Yes. Follow class handle “pointers” to other objects – essentially exploring the OBJECT SPACE that existed at THAT time during simulation. But I’m in post sim!

Can I see all the sequence items that hit my driver? Yes. How? Just put the driver “handle” into the wave window and “open” it. You can see the virtual interface handle (if you have one). You can see the transactions that went through the driver (the driver did a ‘get_next_item (t)’ 100,000 times!).

<Transaction handle ‘t’ from the driver in the wave window, with the driver’s virtual interface>

Driver and 't' in Wave Window

Driver and ‘t’ in Wave Window

In the wave window? Yes. All 100,000 of them? Yes.

Now I’m having fun again. That’s great. I can see what’s going on inside my objects. In post-sim mode.

What’s NOT possible?

Will it babysit? No. One thing at a time.

Are you having fun yet?

Find more details in Verification Horizons article – Old School vs. New School – Visualizer and on Verification Academy – Verification and Debug: Old School Meets New School 

You can find all the sessions on New School verification techniques via the following link:

https://verificationacademy.com/seminars/academy-live

, , , , , ,

17 November, 2014

Few verification tasks are more challenging than trying to achieve code coverage goals for a complex system that, by design, has numerous layers of configuration options and modes of operation.  When the verification effort gets underway and the coverage holes start appearing, even the most creative and thorough UVM testbench architect can be bogged down devising new tests – either constrained-random or even highly directed tests — to reach uncovered areas.

armpaper
At the recent ARM® Techcon, Nguyen Le, a Principal Design Verification Engineer in the Interactive Entertainment Business Unit at Microsoft Corp. documented a real world case study on this exact situation.  Specifically, in the paper titled “Advanced Verification Management and Coverage Closure Techniques”, Nguyen outlined his initial pain in verification management and improving cover closure metrics, and how he conquered both these challenges – speeding up his regression run time by 3x, while simultaneously moving the overall coverage needle up to 97%, and saving 4 man-months in the process!  Here are the highlights:

* DUT in question
— SoC with multi-million gate internal IP blocks
— Consumer electronics end-market = very high volume production = very high cost of failure!

* Verification flow
— Constrained-random, coverage driven approach using UVM, with IP block-level testbenches as well as  SoC level
— Rigorous testplan requirements tracking, supported by a variety of coverage metrics including functional coverage with SystemVerilog covergroups, assertion coverage with SVA covers, and code coverage on statements, Branches, Expressions, Conditions, and FSMs

* Sign-off requirements
— All test requirements tracked through to completion
— 100% functional, assertion and code coverage

* Pain points
— Code coverage: code coverage holes can come from a variety of expected and unforeseen sources: dead code can be due to unused functions in reused IP blocks, from specific configuration settings, or a bug in the code.  Given the rapid pace of the customer’s development cycle, it’s all too easy for dead code to slip into the DUT due to the frequent changes in the RTL, or due to different interpretations of the spec.  “Unexplainably dead” code coverage areas were manually inspected, and the exclusions for properly unreachable code were manually addressed with the addition of pragmas.  Both procedures were time consuming and error prone
— Verification management: the verification cycle and the generated data were managed through manually-maintained scripting.  Optimizing the results display, throughput, and tool control became a growing maintenance burden.

* New automation
— Questa Verification Manager: built around the Unified Coverage Database (UCDB) standard, the tool supports a dynamic verification plan cross-linked with the functional coverage points and code coverage of the DUT.  In this way the dispersed project teams now had a unified view which told them at a glance which tests were contributing the most value, and which areas of the DUT needed more attention.  In parallel, the included administrative features enabled efficient control of large regressions, merging of results, and quick triage of failures.

— Questa CoverCheck: this tool reads code coverage results from simulation in UCDB, and then leverages formal technology under-the-hood to mathematically prove that no stimulus could ever activate the code in question. If it’s OK for a given block of code to be dead due to a particular configuration choice, etc., the user can automatically generate wavers to refine the code coverage results.  Additionally, the tool can also identify segments of code that, though difficult to reach, might someday be exercised in silicon. In such cases, CoverCheck helps point the way to testbench enhancements to better reach these parts of the design.

— The above tools used in concert (along with Questasim) enabled a very straightforward coverage score improvement process as follows:
1 – Run full regression and merge the UCDB files
2 – Run Questa CoverCheck with the master UCDB created in (1)
3 – Use CoverCheck to generate exclusions for “legitimate” unreachable holes, and apply said exclusions to the UCDB
4 – Use CoverCheck to generate waveforms for reachable holes, and share these with the testbench developer(s) to refine the stimulus
5 – Report the new & improved coverage results in Verification Manager

* Results
— Automation with Verification Manager enabled Microsoft to reduce the variation of test sequences from 10x runtime down to a focused 2x variation.  Additionally, using the coverage reporting to rank and optimize their tests, they increased their regression throughput by 3x!
— With CoverCheck, the Microsoft engineers improved code coverage by 10 – 15% in most hand-coded RTL blocks, saw up to 20% coverage improvement for auto-generated RTL code, and in a matter of hours were able to increase their overall coverage number from 87% to 97%!
— Bottom-line: the customer estimated that they saved 4 man-months on one project with this process

2014 MSFT presentation at ARM Techcon -- cover check ROI

Taking a step back, success stories like this one, where automated, formal-based applications leverage the exhaustive nature of formal analysis to tame once intractable problems (which require no prior knowledge of formal or assertion-based verification), are becoming more common by the day.  In this case, Mentor’s formal-based CoverCheck is clearly the right tool for this specific verification need, literally filling in the gaps in a traditional UVM testbench verification flow.  Hence, I believe the overall moral of the story is a simple rule of thumb: when you are grappling with a “last mile problem” of unearthing all the unexpected, yet potentially damaging corner cases, consider a formal-based application as the best tool for job.  Wouldn’t you agree?

Joe Hupcey III

Reference links:

Direct link to the presentation slides: https://verificationacademy.com/Advanced-Verification-Management-Presentation

ARM Techcon 2014 Proceedings: http://www.armtechcon.com/

Official paper citation:
Advanced Verification Management and Coverage Closure Techniques, Nguyen Le, Microsoft; Harsh Patel, Roger Sabbagh, Darron May, Josef Derner, Mentor Graphics

, , , , , , , , , , ,

@dennisbrophy Tweets

  • Loading tweets...

@dave_59 Tweets

  • Loading tweets...

@jhupcey Tweets

  • Loading tweets...