Posts Tagged ‘Assertion-Based Verification’
Prologue: The 2012 Wilson Research Group Functional Verification Study
This is the first in a series of blogs that presents the results from the 2012 Wilson Research Group Functional Verification Study.
Study Overview
In 2002 and 2004, Ron Collett International, Inc. conducted its well known ASIC/IC functional verification studies, which provided invaluable insight into the state of the electronic industry and its trends in design and verification. However, after the 2004 study, no other industry studies were conducted, which left a void in identifying industry trends.
To address this void, Mentor Graphics commissioned Far West Research to conduct an industry study on functional verification in the fall of 2007. Then in the fall of 2010, Mentor commissioned Wilson Research Group to conduct another functional verification study. Both of these studies were conducted as blind studies to avoid influencing the results. This means that the survey participants did not know that the study was commissioned by Mentor Graphics. In addition, to support trend analysis on the data, both studies followed the same format and questions (when possible) as the original 2002 and 2004 Collett studies.
In the fall of 2012, Mentor Graphics commissioned Wilson Research Group again to conduct a new functional verification study. This study was also a blind study and follows the same format as the Collett, Far West Research, and previous Wilson Research Group studies. The 2012 Wilson Research Group study is one of the largest functional verification studies ever conducted. The overall confidence level of the study was calculated to be 95% with a margin of error of 4.05%.
Unlike the previous Collett and Far West Research studies that were conducted only in North America, both the 2010 and 2012 Wilson Research Group studies were worldwide studies. The regions targeted were:
- North America:Canada,United States
- Europe/Israel:Finland,France,Germany,Israel,Italy,Sweden,UK
- Asia (minusIndia):China,Korea,Japan,Taiwan
- India
The survey results are compiled both globally and regionally for analysis.
Another difference between the Wilson Research Group and previous industry studies is that both of the Wilson Research Group studies also included FPGA projects. Hence for the first time, we are able to present some emerging trends in the FPGA functional verification space.
Figure 1 shows the percentage makeup of survey participants by their job description. The red bars represents the FPGA participants while the green bars represent the non-FPGA (i.e., IC/ASIC) participants.
Figure 1: Survey participants job title description
Figure 2 shows the percentage makeup of survey participants by company type. Again, the red bars represents the FPGA participants while the green bars represents the non-FPGA (i.e., IC/ASIC) participants.
Figure 2: Survey participants company description
In a future set of blogs, over the course of the next few months, I plan to present the highlights from the 2012 Wilson Research Group study along with my analysis, comments, and obviously, opinions. A few interesting observations emerged from the study, which include:
- FPGA projects are beginning to adopt advanced verification techniques due to increased design complexity.
- The effort spent on verification is increasing.
- The industry is converging on common processes driven by maturing industry standards.
My next blog presents current design trends that were identified by the survey. This will be followed by a set of blogs focused on the functional verification results.
Also, to learn more about the 2012 Wilson Reserach Group study, view my pre-recorded Functional Verification Study web-seminar, which is located out on the Verification Academy website.
Quick links to the 2012 Wilson Research Group Study results (so far…)
- Part 1 – Design Trends
Tags: accellera, Assertion-Based Verification, formal verification, functional coverage, functional verification, IEEE, Simulation, Standards, SystemVerilog, UVM, Verification Academy, Verification Methodology, verilog, vhdl
Synthesizing Hardware Assertions and Post-Silicon Debug
At the 2012 Design Automation Conference, I had the pleasure of moderating a panel at a workshop titled “Post-Silicon Debug: Technologies, Methodologies, and Best-Practices.” This workshop brought together a collection of experts from industry, academia, and EDA to discuss the emerging challenges and solutions associated with post-silicon validation. The speakers presented different instrumentation strategies, as well as methods of using data collected by the debug logic to facilitate fast and efficient debug.
Performing verification on real silicon introduces a number of new and unique challenges. On the one hand, real silicon offers great execution speed, which enables a long test run that reaches deep into the design’s state-space. On the other hand, real silicon lacks both good controllability and observability, which serve an important role in pre-silicon verification. Assertions, which have always been one of my passions, have been shown to address both the controllability and observability challenges associated pre-silicon verification (for example, RTL simulation). And now, there is emerging interest in addressing these same challenges in post-silicon validation.
I’d like to invite you to check out my Tech Design Forum article titled Synthesizing assertion into hardware for faster debug. Obviously, synthesizing hardware assertions is only one of many new solutions that are currently being explored to contain the growing cost and effort associated with post-silicon debug. One attractive benefit of assertion-based techniques is that they provide a nice natural link between pre-silicon verification and post-silicon validation, in terms of reuse.
I’d like to hear your opinions concerning synthesizing hardware assertions, as well as post-silicon debugging challenges in general.
Tags: Assertion-Based Verification, Validation, Verification
Verification Academy: Up Close & Personal
Live & In-Person at DAC 2012!
Verification Academy, the brain child of Harry Foster, Chief Verification Scientist at Mentor Graphics, was live from the Design Automation Conference tradeshow floor this year. Harry is pictured to the right giving an update on his popular verification survey from the DAC tradeshow floor.
The Verification Academy, predominantly a web-based resource is a popular site for verification information with more than 11,000 registered members for forum access on topics ranging from OVM/UVM, SystemVerilog and Analog/Mixed-Signal design. The popular OVM/UVM Cookbook, which used to be available as a print edition, is now a live online resource there as well. A whole host of educational modules and seminars can also be found there too.
If you know about the Verification Academy, you know all about the content mentioned above and that there is much more to be found there. For those who don’t know as much about it, Harry took a break from the being at the Verification Academy booth at DAC to discuss the Verification Academy with Luke Collins, Technology Journalist, Tech Design Forum. (Flash is required to watch Harry discuss Verification Academy with Luke.)
The Verification Academy at DAC was a great venue to connect in person with other Verification Academy users to discuss standards, methodologies, flows and other industry trends. Each hour there were short presentations by Verification Academy members that proved to be a popular way to start some interesting conversations. While we realize not all Verification Academy members were able to attend DAC in person, we know many have expressed an interest to some of the presentations. Verification Academy “Total Access” members now have access to many of the presentations.
|
ARM |
||
|
Doulos |
||
|
Thales Alenia Space |
||
|
Test & Verification Solutions |
||
|
Willamette HDL |
||
|
Sunburst Design |
||
|
Mentor Graphics |
Total Access members can also download all the presentations in a .zip file. Happy reading to all those who were unable to visit us at DAC and thank you to all who were able to stop by and visit.
Tags: ABV, ACE, ams, ARM, Assertion-Based Verification, Coverage Closure, dac, Doulos, formal, IEEE, iTBA, Low Power, OVM, SystemVerilog, Tech Design Forum, Thales, upf, UVM, UVM Express, Verification Academy, Verification Trends
Part 8: The 2010 Wilson Research Group Functional Verification Study
Language and Library Trends
This blog is a continuation of a series of blogs, which present the highlights from the 2010 Wilson Research Group Functional Verification Study (for a background on the study, click here).
In my previous blog (Part 7 click here), I focused on some of the 2010 Wilson Research Group findings related to testbench characteristics and simulation strategies. In this blog, I present design and verification language trends, as identified by the Wilson Research Group study.
You might note for some of the language and library data I present, the percentage sums to more than one hundred percent. The reason for this is that some perticipant’s projects use multiple languages and multiple methodologies.
Design Languages
Let’s begin by examining the languages used for design, as shown in Figure 1. Here, we compare the results for languages used to design FPGAs (in grey) with languages used to design non-FPGAs (in green).
Figure 1. Languages used for design
Not too surprising, we see that VHDL is the most popular language used for the design of FPGAs, while Verilog and SystemVerilog are the most popular languages used for the design of non-FPGAs.
Figure 2 shows the trends in terms of languages used for design, by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green), as well as the projected design language adoption trends within the next twelve months (in purple). Note that the design language adoption is declining for most of the languages with the exception of SystemVerilog whose adoption is increasing.
Figure 2. Trends in languages used for design
Verification Languages
Next, let’s look at the languages used for verification (that is, languages used to create simulation testbenches). Figure 3 compares the results between FPGA designs (in grey) and non-FPGA designs (in green). 
Figure 3. Languages used in verification to create simulation testbenches
And again, it’s not too surprising to see that VHDL is the most popular language used to create verification testbenches for FPGAs, while SystemVerilog is the most popular language used to create testbenches for non-FPGAs.
Figure 4 shows the trends in terms of languages used to create simulation testbenches by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green), as well as the projected language adoption trends within the next twelve months (in purple). Note that verification language adoption is declining for most of the languages with the exception of SystemVerilog whose adoption is increasing.
Figure 4. Trends in languages used in verification to create simulation testbenches
Now, let’s look at methodology and class library adoption. Figure 5 shows the future trends in terms of methodology and class library adoption by comparing the 2010 Wilson Research Group study (in green) with the projected adoption trends within the next twelve months (in purple). Previous studies did not include data on methodology and class library adoption, so we are unable to show previous trends.
Figure 5. Methodology and class library future trends
The study indicates that the only methodology adoption projected to grow in the next twelve months are OVM and UVM.
Assertion Languages and Libraries
Finally, let’s examine assertion language and library adoption, as shown in Figure 6. Here, we compare the results for FPGA designs (in grey) and non-FPGA designs (in green).
Figure 6. Assertion language and library adoption
SystemVerilog Assertions (SVA) is the most popular assertion language used for both FPGA and non-FPGA designs.
Figure 7 shows the trends in terms assertion language and library adoption by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green), as well as the projected adoption trends within the next twelve months (in purple). Note that the adoption of most of the assertion languages is declining, with the exception of SVA whose adoption is increasing.
Figure 7. Trends in assertion language and library adoption
In my next blog (click here), I plan to focus on the adoption of various verification technologies and techniques used in the industry, as identified by the 2010 Wilson Research Group study.
Tags: 1076, 1364, 1666, 1800, accellera, Add new tag, Assertion-Based Verification, functional verification, IEEE 1800, OVM, Standards, SystemVerilog, UVM, Verification Methodology, verilog, vhdl, vmm
The Survey Says: Verification Planning
As the saying goes: Those who fail to plan, plan to fail. With that said, I am excited to announce a new module focused on Verification Planning, which has been one of the Verification Academy’s most-requested subjects for new content. The new Verification Planning module is delivered by our subject matter expert, who literally wrote the book on the subject, Peet James.
The goal of verification planning and management is to architect an overall verification approach, and then to document that approach in a family of useful, easily extracted, maintainable verification documents that will strategically guide the overall verification effort so that the most amount of verification is accomplished in the allotted time. The aim of this module is to define terms, logically divide up the verification effort, and lay the foundation for actual verification planning and management on a real project.
I think you will really enjoy and be enlightened by Peet’s treatment of the subject, and hopefully, you can apply many of the techniques that he presents to your own projects.
Speaking of applying Verification Academy techniques—we just conducted a large survey about the academy and found some interesting results that I would like to share with you. First, Figure 1 shows who is viewing the Verification Academy content by job title.
Figure 1: Verification Academy viewers by job title
It’s not too surprising that a majority of the viewers are verification engineers, with a ratio of about 3.5 verification engineers for every 2 designers.
In addition to who is viewing the Verification Academy, we were interested in learning the viewer’s type of targeted design implementation to get a better understanding of our viewers’ needs. Figure 2 shows who is viewing the Verification Academy by their type of targeted design implementation.
Figure 2: Verification Academy viewers by targeted design implementation
We are obviously seeing a growing number of FPGA engineers interested in advanced functional verification. Today’s complex SoC-base FPGA designs are not your mom and pop variety of FPGA designs. More advanced verification skills are required to ultimately meet both quality and schedule goals.
Another question we wanted to answer through our survey is whether the Verification Academy has been useful. One way to answer this is to see how many viewers had actually applied or plan to apply the knowledge they learned in the Verification Academy on their own projects. The survey results are shown in Figure 3.
Figure 3: Verification Academy viewers who have applied knowledge on projects
We also wanted to determine through the survey if the content presented in the Verification Academy was at an appropriate level of detail. The survey results are shown in Figure 4.
Finally, we wanted to determine through the survey which additional topic in advanced functional verification should be covered in the Verification Academy. Figure 5 presents the results.
Figure 5: Verification Academy new subject content request
Your feedback is important to us, and we are very excited that our new Verification Planning module was one of the top requests from the Verification Academy survey participants.
I would like to encourage you to check out all our new and existing content at the Verification Academy by visiting www.verificationacademy.com.
Tags: Assertion-Based Verification, functional verification, OVM, UVM, Verification Academy, Verification Methodology
Making formal property checking easy to use
For years one of the objectives in EDA has been to make formal property checking easy to use and its results easy to understand. With the Automatic formal check feature in the June release of the 0-In Formal tool version 3.0, I think we have made significant progress in this area.
The feature, which predefines a set of assertion rules to look for design issues automatically, makes formal technology accessible to users who are not yet ready to write properties in System Verilog Assertion (SVA) or Property Specification Language (PSL). To make it easier to comprehend problems in the design, the tool highlights the violations back to the RTL code.
Automatic formal check focuses on three areas inadequately addressed by dynamical simulation:
The first area is functional coverage. Today, when constrained random simulation fails to achieve the targeted coverage goal, engineers have to fine tune the environment or add new tests. These efforts, often attempted relatively late in the verification cycle, can consume vast amounts of time and resources while still failing to reach parts of the design. In contrast, automatic formal check can be used to identify unreachable code early in the verification cycle. These targets can be eliminated from the coverage model. As a result, the coverage measurement is more accurate and you know when you are done.
The next area is design initialization. If a design cannot be initialized reliably in silicon, it will not function correctly. An obvious precursor then is making sure all the registers are initialized correctly at RTL. If X’es are used, we need to monitor the X creation, propagation and usage cycle. Dynamic simulation does not interpret X’es accurately as in silicon, which has only 1s and 0s. Automatic formal check is ideal in verifying register initialization under different modes or configurations. Then, with internal assertions and formal technologies, we can check that although X’es are created, they are not used by downstream registers.
The final area is corner case design issues. Time and time again, designers unintentionally write code that violates logical correctness. Examples include combinational loops, full case violations, parallel case violations, undriven logic, finite-state machine (FSM) deadlocks and FSM livelocks. Unless tests are written to specifically target these corner case design issues are, such issues are difficult to exercise. On the other hand, by formally analyzing the design semantics, automatic formal check identifies these design issues statically and creates the stimuli to highlight them to the users.
If you are interested to know more about the automatic formal check feature in 0-In Formal, please feel free to register for our upcoming seminar in San Jose.
Tags: Assertion-Based Verification, automatic formal check, formal property checking, formal verification, functional coverage, RTL, SVA
Static Verification
After spending years verifying ASICs with dynamic simulation, I started working on static verification 10 years ago in a startup called 0-In Design Automation. I firmly believe that static verification can complement dynamic simulation. Static verification uses synthesis and formal technologies to find bugs in the design. It does not rely on simulation stimulus. You do not need to exercise the bugs, propagate the results, and check the outputs to detect them.
Static verification includes RTL lint, static checks, formal checks, automated and assertion-based formal property checking. To read more on static verification, you can take a look at my white paper: Getting Started With Static Verification. If you are interested in formal methods, you can take a look at Harry Foster’s white paper: Why Now for Formal Property Checking. Both can be found in the Knowledge Center of DAC.com.
In the future, we are going to talk about individual static verification technologies and its application in areas such as RTL verification, clock domain crossing verification, low power verification, timing constraint verification, etc. Your feedback and comments are most welcome.
Tags: 0-In, Assertion-Based Verification, CDC, dac, formal, RTL, Verification
DAC Panel: Bridging Pre-Silicon Verification and Post-Silicon Validation
I’d like to encourage you to attend the technical panel titled Bridging Pre-Silicon Verification and Post-Silicon Validation at this year’s DAC. The panel will be held on Tuesday, June 15, 2010 between 2:00 PM—4:00 PM.
|
Chair: |
Alan Hu – Univ. of British Columbia, Vancouver, BC, Canada |
|
Organizers: |
Rajesh Galivanche – Intel Corp., Santa Clara, CA |
|
|
Amir Nahir – IBM Corp., Haifa, Israel |
|
|
Avi Ziv – IBM Corp., Haifa, Israel |
|
|
|
|
Speakers: |
Miron Abramovici – Tiger’s Lair, Inc., Vienna, VA |
|
|
Sean Baartmans – Intel Corp., Hillsboro, OR |
|
|
Valeria Bertacco – Univ. of Michigan, Ann Arbor, MI |
|
|
Albert Camilleri – Qualcomm, Inc., San Diego, CA |
|
|
Harry Foster – Mentor Graphics Corp., Plano, TX |
|
|
Shakti Kapoor – IBM Corp., Austin, TX |
Why do I think this is an important topic? At 65nm, we witnessed the post-silicon validation effort often consuming more than 50% of an SoC’s overall design effort, as measured in cost, and the problem grows as the industry continues to move to even smaller geometries. Unlike pre-silicon verification, which has historically (and conveniently) partitioned the verification effort into separate concerns (such as, electrical, functional, performance, and software), identifying failures in post-silicon requires skills spanning multiple validation disciplines. Furthermore, the process of post-silicon validation is hindered by both poor observability and poor controllability. To address today’s escalating validation effort requires establishing a stronger link between the pre-silicon verification and post-silicon validation processes. Certainly assertions are one technique that can bridge pre- and post-silicon by providing improved observability on critical functionality. However, the improvements obtained by silicon assertions are only as effective as the quality of their pre-silicon form. Realistically, only a small set of critical assertions could be shared between the pre- and post-silicon processes. What is needed is a means to instrument into the silicon observability in a reconfigurable fashion, thus allowing the post-silicon validation engineer to shift focus on specific areas of concern. Concerning test generation, pre-silicon test provides insufficient coverage to stress the post-silicon design. Hence, what is needed is a means to capture post-silicon test associated with a failure in an abstract form that can be demonstrated on a pre-silicon model. Finally, concerning triage and error isolation, both the pre-silicon verification and post-silicon validation processes could benefit from automatic techniques that identify a set of candidate causes behind the detected failure.
For more information about the upcoming panel, visit: http://www2.dac.com/panels.aspx?event=30&topic=11
A Cliffhanger ABV Seminar, Jan 19, Santa Clara, CA
I’m excited. I’ve had the pleasure of knowing Cliff Cummings for many years, and I was honored a couple of years ago to have him write the foreword in a book that I published on assertions. Now, we have joined forces to do a set of seminars titled: “Assertion-Based Verification for FPGA and IC Design.” The first seminar will take place on January 19, 2010 in Santa Clara, CA, and you can register online by clicking here.
This six-hour seminar is organized into four sessions. My session is titled: “Industry Perspective and Opportunities in Assertion-Based Verification,” and I intend to provide a survey of today’s ABV landscape, ranging from various industry myths to realities. In fact, I specifically plan on addressing the issues raised in my recent blog titled “Evolution is a tinkerer.” In addition, I’ll talk about what characterizes a successful organization that has successfully adopted ABV, and then contrast it against organizations that are struggling or have failed in their attempt to integrate ABV into their flow.
The second and third sessions will focus on “Advanced Debugging with Assertions” and “Effective Coverage Using Assertions.”
We will conclude the seminar with an extended session covering “Basic SystemVerilog Assertions Training” by our SystemVerilog guru Cliff Cummings! His presentation details practical SystemVerilog assertion tricks that you can apply today to your own work, as well as methodological recommendations to improve efficiency when adopting ABV.
I hope everyone has a peaceful, happy holiday, and I look forward to meeting you on January 19 at the ABV seminar in Santa Clara, CA!
Tags: Add new tag, Assertion-Based Verification, SystemVerilog, Verification Academy
Evolution is a tinkerer
I was recently quoted in an EDA DesignLine blog as saying that “it is a myth that ABV is a mainstream technology.” Actually, the original quote comes from an extended abstract I wrote for an invited tutorial at Computer-Aided Engineering (CAV) in 2008 titled Assertion-Based Verification: Industry Myths to Realities. My claim is based on the Farwest Research 2007 study (comissioned and sponsored by Mentor Graphics) that found approximately 37 percent of the industry had adopted simulation-based ABV techniques, and 19 percent of the industry had adopted formal ABV techniques. Now, those of you who know me know that I am an optimist—and therefore the statistics from these industry studies reveal a wonderful opportunity for design projects to improve themselves. 
However, the problem of adopting advanced functional verification techniques is not just limited to ABV. For example, the study revealed that only 48 percent of the industry performs code coverage. Let me repeat that, I said code coverage, not even something as exotic as functional coverage (which was observed to be about 40 percent of the industry)! Furthermore, only about 41 percent of the industry has adopted constrained random verification. 
Now, we could argue about which is the best approach to measuring coverage or achieving functional closure, but that is not the point. The question is, how do we as an industry evolve our verification capabilities beyond 1990 best practices?
In my mind, one of the first steps in the evolutionary process is to define a model for assessing an organization’s existing verification capabilities. For a number of years I’ve studied variants of the Capability Maturity Model Integration (that is, CMM and CMMI) as a possible tool for assessment. After numerous discussions with many industry thought leaders and experts, I’ve concluded that the CMM is really not an ideal model for assessing hardware organizations. Nonetheless, there is certainly a lot we can learn from observing CMM applied on actual software projects.
For those of you unfamiliar with the CMM, its origins date back to the early 1980’s. During this period, the United States Department of Defense established the Software Engineering Institute at Carnegie Mellon University in response to a perceived software development crisis related to escalating development costs and quality problems. One of the key contributions resulting from this effort was the published work titled The Capability Maturity Model: Guidelines for Improving the Software Process. The CMM is a framework for assessing effective software process, which provides an evolutionary path for improving an organization’s processes from ad hoc, immature processes to developed, mature, disciplined ones.
Fundamental to maturing an organization’s process capabilities is an investment in developing skills within the organization. To assist in this effort, we have launched an ambitious project to evolve an organization’s advanced functional verification skills through the Verification Academy. In fact, we have an introductory module titled Evolving Capabilities that provides my first attempt at a simple assessment model. I anticipate that this model will itself evolve over time as I receive valuable feedback on refining and improving it. Nonetheless, the simple Evolving Capabilities model as it exists today provides a wonderful framework for organizing multiple modules focused on evolving an organization’s advanced functional verification capabilities.
I realize that evolving technical skills is obviously only part of the solution to successfully advancing the industry’s functional verification capabilities. Yet, education is an important step. I’d be interested in hearing your thoughts on the subject. Why do you think the industry as a whole has been slow in its adoption of advanced functional verification techniques? What can be done to improve this situation?
About Verification Horizons BLOG
This blog will provide an online forum to provide weekly updates on concepts, values, standards, methodologies and examples to assist with the understanding of what advanced functional verification technologies can do and how to most effectively apply them. We're looking forward to your comments and suggestions on the posts to make this a useful tool.
Latest Posts
- Texas-Sized DAC Edition of Verification Horizons Now Up on Verification Academy
- IEEE 1801™-2013 UPF Standard Is Published
- Part 1: The 2012 Wilson Research Group Functional Verification Study
- What’s the deal with those wire’s and reg’s in Verilog
- Getting AMP’ed Up on the IEEE Low-Power Standard
- Prologue: The 2012 Wilson Research Group Functional Verification Study
Subscribe to the Verification Horizons Newsletter