Archive for Harry Foster
Verification Horizons BLOG
Part 1: The 2012 Wilson Research Group Functional Verification Study
Design Trends
In my previous blog, I introduced the 2012 Wilson Research Group Functional Verification Study (click here). The objective of my previous blog was to provide background on this large, worldwide industry study. I will present the key findings from this study in a set of upcoming blogs.
This blog begins the process of revealing the 2012 Wilson Research Group study findings by first focusing on current design trends. Let’s begin by examining process geometry adoption trends, as shown in Figure 1. Here, you will see trend comparisons between the 2007 Far West Research study (gray line), the 2010 Wilson Research Group study (blue line), and the 2012 Wilson Research Group study (green line).
Figure 1. Process geometry trends
Worldwide, the median process geometry size from the 2007 Far West Research study was about 90nm, while the median process geometry size is about 65nm in 2010. Today, the mean process geometry size for a typical project is about 45nm—although you can see that over a third of projects today are designing below 32nm.
In addition to the industry moving to smaller process geometries, the industry is also moving to larger design sizes as measured in number of gates of logic and datapath, excluding memories (which should not be a surprise). Figure 2 compares design sizes from the 2002 Collett study (dark blue line), the 2007 Far West Research study (gray line), the 2010 Wilson Research Group study (light blue line), and the 2012 Wilson Research Group study (green line).
Figure 2. Number of gates of logic and datapath trends, excluding memories
The study revealed that about a third of the non-FPGA designs today are less than 5M gates, while a third range in size between 5M to 20M gates, and about a third of all designs are larger than 20M gates.
It’s important to note here that the data on the mean design size trends does not reflect volume in terms of semiconductor production. For example, you could have fewer projects designing at a small geometry, yet they have higher volume in terms of production.
In Figure 3, I show the mean design size trends between the 2002 Collett study (dark blue line), the 2007 Far West Research study (gray line), the 2010 Wilson Research Group study (light blue line), and the 2012 Wilson Research Group study (green line). Obviously, gate counts have increased over the years, yet a significant number of designs continue to be developed with smaller (and larger) gate counts as indicated by the mean calculation. Another observation is that, as you would expect, the mean gate count trend is essentially following Moore’s law.
Figure 3. Mean design size trends
Figure 4 presents the current design implementation trends for non-FPGAs as identified by the survey participants.
Figure 4. Non-FPGA current design implementation trends
The data in Figure 4 presents trends in design implementation approaches for non-FPGA designs, ranging from the 2002 Collett study (dark blue bar), the 2004 Collet study (dark green bar), the 2007 Far West Research study (gray bar), the 2010 Wilson Research Group study (blue bar), and the 2012 Wilson Research Group study (green bar). Note that the study seems to indicate that there is a downward trend in standard cell design implementation.
Figure 5. FPGA design implementation trends
For the 2012 study, we decided that we wanted to get a sense of the percentage of FPGA projects that target the very complex programmable SoC FPGAs that have recently emerged, which is shown in Figure 5. Examples of these programmable SoC FPGAs include: Xilinx’s Zynq, Altera’s Arria/Cydone, and Microsemi’s SmarFusion.
In my next blog (click here), I’ll continue discussing current design trends, focusing specifically on embedded processors, power, and clock domains.
Tags: accellera, functional verification, IEEE, IEEE 1800, Verification, Verification Academy
Prologue: The 2012 Wilson Research Group Functional Verification Study
This is the first in a series of blogs that presents the results from the 2012 Wilson Research Group Functional Verification Study.
Study Overview
In 2002 and 2004, Ron Collett International, Inc. conducted its well known ASIC/IC functional verification studies, which provided invaluable insight into the state of the electronic industry and its trends in design and verification. However, after the 2004 study, no other industry studies were conducted, which left a void in identifying industry trends.
To address this void, Mentor Graphics commissioned Far West Research to conduct an industry study on functional verification in the fall of 2007. Then in the fall of 2010, Mentor commissioned Wilson Research Group to conduct another functional verification study. Both of these studies were conducted as blind studies to avoid influencing the results. This means that the survey participants did not know that the study was commissioned by Mentor Graphics. In addition, to support trend analysis on the data, both studies followed the same format and questions (when possible) as the original 2002 and 2004 Collett studies.
In the fall of 2012, Mentor Graphics commissioned Wilson Research Group again to conduct a new functional verification study. This study was also a blind study and follows the same format as the Collett, Far West Research, and previous Wilson Research Group studies. The 2012 Wilson Research Group study is one of the largest functional verification studies ever conducted. The overall confidence level of the study was calculated to be 95% with a margin of error of 4.05%.
Unlike the previous Collett and Far West Research studies that were conducted only in North America, both the 2010 and 2012 Wilson Research Group studies were worldwide studies. The regions targeted were:
- North America:Canada,United States
- Europe/Israel:Finland,France,Germany,Israel,Italy,Sweden,UK
- Asia (minusIndia):China,Korea,Japan,Taiwan
- India
The survey results are compiled both globally and regionally for analysis.
Another difference between the Wilson Research Group and previous industry studies is that both of the Wilson Research Group studies also included FPGA projects. Hence for the first time, we are able to present some emerging trends in the FPGA functional verification space.
Figure 1 shows the percentage makeup of survey participants by their job description. The red bars represents the FPGA participants while the green bars represent the non-FPGA (i.e., IC/ASIC) participants.
Figure 1: Survey participants job title description
Figure 2 shows the percentage makeup of survey participants by company type. Again, the red bars represents the FPGA participants while the green bars represents the non-FPGA (i.e., IC/ASIC) participants.
Figure 2: Survey participants company description
In a future set of blogs, over the course of the next few months, I plan to present the highlights from the 2012 Wilson Research Group study along with my analysis, comments, and obviously, opinions. A few interesting observations emerged from the study, which include:
- FPGA projects are beginning to adopt advanced verification techniques due to increased design complexity.
- The effort spent on verification is increasing.
- The industry is converging on common processes driven by maturing industry standards.
My next blog presents current design trends that were identified by the survey. This will be followed by a set of blogs focused on the functional verification results.
Also, to learn more about the 2012 Wilson Reserach Group study, view my pre-recorded Functional Verification Study web-seminar, which is located out on the Verification Academy website.
Quick links to the 2012 Wilson Research Group Study results (so far…)
- Part 1 – Design Trends
Tags: accellera, Assertion-Based Verification, formal verification, functional coverage, functional verification, IEEE, Simulation, Standards, SystemVerilog, UVM, Verification Academy, Verification Methodology, verilog, vhdl
Improving simulation results with formal-based technology
When it comes to formal methods, many engineers are skeptics. Perhaps this is due to value propositions that have been pitched over the years that have over-promised yet under-delivered in terms of results. Or perhaps it is due to the advanced skills that have traditionally been required to achieve predictable and reliable results. After all, historically this was the case—dating back to the mid-nineties when formal techniques were only adopted by companies that could afford a dedicated team of formal experts with PhDs.
So, what’s changed today? The emergence of functional verification solutions targeted at specific problem domains, which blend simulation with formal-based techniques in a seamless way to improve results. In other words, the application of formal-based technology is not just for experts anymore! In fact, everyone can reap the benefits of formal analysis today with very little effort.
One example of this blending of simulation with formal-based techniques is in the area of accelerating the process of code coverage closure with the new Questa CoverCheck solution. Closing code coverage typically involves many engineering weeks of effort to manually review code coverage holes to determine if they are unreachable and can be safely ignored—or figuring out exactly how to handcraft special tests to cover them during simulation. Questa CoverCheck makes it easy for non-expert users to leverage formal-based technology to complete this process by automatically identifying the set of unreachable coverage items in a design, and then guiding the user to create tests for the reachable items that have not been covered yet. This process, illustrated in the figure below, is push-button, low-effort, and requires no expertise with formal techniques. In addition, no assertions are required nor expertise in assertion languages. It is a beautiful example of how formal-based technology is blended with simulation to form a solution that improves both productivity and quality of results.
Another example of how formal-based technology is being used today to complement simulation is with AutoCheck, which is part of the Questa Formal solution. For example, there is a class of bugs that cannot be found using RTL simulation due to a simulation effect known as X-state optimism. These bugs might be found during gate-level simulation, but this occurs very late in the design flow when it is costlier to fix. By using AutoCheck, engineers are able to identify and correct X-state issues early in the design flow, before simulation occurs. In addition to X-state issues, AutoCheck uses formal-based technology to verify a wide range of common RTL errors that are difficult or impossible to find during RTL simulation. It is another example of a push-button, low-effort solution where assertion-language and formal expertise is not required. What’s new in the latest Questa Formal release is significant improvements in engine performance and capacity, along with multicore support.
Questa CDC is one more example of how formal-based technology is being used today to complement simulation. Today, we see about 94% of all designs have multiple asynchronous clock domains. Verifying that a signal originating from one clock domain will safely be registered in a different asynchronous clock domain is not possible using traditional RTL simulation since state element setup and hold times are not modeled, which means that metastability issues will not be verified. Again, these bugs might be found later in the flow during gate-level simulation where it is costlier to fix. Static timing analysis, although effective at finding timing issues within a single or synchronous clock domains is unable to identify issues across asyncrhonous clock domains. This is an area with formal-based technology, such as Questa CDC, can help. What’s new in the latest Questa CDC release is support for unlimited design sizes through hierarchical CDC analysis along with a 5X improvement in performance.
To learn what’s new with Questa Formal-Based Technology, see our recent press release. Or check out the Questa Cover Check, Questa CDC and Questa Formal links.
Tags: formal verification, functional verification, Simulation
Synthesizing Hardware Assertions and Post-Silicon Debug
At the 2012 Design Automation Conference, I had the pleasure of moderating a panel at a workshop titled “Post-Silicon Debug: Technologies, Methodologies, and Best-Practices.” This workshop brought together a collection of experts from industry, academia, and EDA to discuss the emerging challenges and solutions associated with post-silicon validation. The speakers presented different instrumentation strategies, as well as methods of using data collected by the debug logic to facilitate fast and efficient debug.
Performing verification on real silicon introduces a number of new and unique challenges. On the one hand, real silicon offers great execution speed, which enables a long test run that reaches deep into the design’s state-space. On the other hand, real silicon lacks both good controllability and observability, which serve an important role in pre-silicon verification. Assertions, which have always been one of my passions, have been shown to address both the controllability and observability challenges associated pre-silicon verification (for example, RTL simulation). And now, there is emerging interest in addressing these same challenges in post-silicon validation.
I’d like to invite you to check out my Tech Design Forum article titled Synthesizing assertion into hardware for faster debug. Obviously, synthesizing hardware assertions is only one of many new solutions that are currently being explored to contain the growing cost and effort associated with post-silicon debug. One attractive benefit of assertion-based techniques is that they provide a nice natural link between pre-silicon verification and post-silicon validation, in terms of reuse.
I’d like to hear your opinions concerning synthesizing hardware assertions, as well as post-silicon debugging challenges in general.
Tags: Assertion-Based Verification, Validation, Verification
Expanding the Verification Academy!
The philosophy behind our Verification Academy is to provide a comprehensive resource for evolving and maturing your functional verification process skills. We believe that each step an organization takes in evolving verification skills should have measurable results and benefits. With that in mind, I am excited to announce two new modules we are adding to the Verification Academy: UVM Express and Advanced UVM.
Figure 1. Screen shot for the Verification Academy UVM Epxress module.
For some verification teams, the hurdle to implement a UVM-based verification environment is simply getting started. To eliminate this hurdle, Mentor Graphics has just introduced UVM Express, which is a suggested systematic way to progressively adopt a UVM methodology. The beauty of this approach is that UVM Express makes getting started easy and intuitive, and provides measurable productivity gains to a broader scope of design projects, regardless of their past experience and existing skills with UVM.
Our new UVM Express module consists of four sessions for evolving UVM capability. It is not necessary for a project to implement all the recommendations from the four sessions to receive measurable benefits. For example, a project might decide to only adopt the recommendations presented in the first session in the UVM Express module, and then at some future point, decide to evolve their skills further by adopting the recommendations from the second session.
In addition to the UVM Express module, we have released an Advanced UVM module. This module consists of 10 sessions and provides over three hours of material. In this module, we build on the concepts originally covered in our Basic UVM Module to take your UVM understanding to the next level. You will learn how to build tests and verification environments, use the factory and configuration database to customize your verification IP, and create reusable stimulus sequences, including those for multi-layer protocols. We will also introduce the UVM Register layer, showing you how to create a register model and how to write and reuse register-level tests.
We are excited to now offer you three UVM modules within the Verification Academy: Basic UVM, Advanced UVM, and UVM Express. And as always, I welcome your feedback and suggestions.
For more information, visit http://www.verificationacademy.com/.
Tags: UVM, Verification Academy
Verification solutions that help reduce bug cost
I think very few engineers would argue with the claim that the longer a bug goes undetected, the more expensive it is to fix. In fact, the general rule-of-thumb is that the cost to fix a bug increases by an order of magnitude as a project progresses from one milestone to the next. Bugs found before simulation obviously have the lowest cost. Bugs found at block or subsystem simulation are generally easier to debug and incur a lower cost to fix versus bugs found at high levels of integration—such as chip- and system-level simulation. Bugs found during post-silicon validation might require a new spin of the chip, while bugs found after product release can be devastating, costing millions of dollars to fix.
Years ago, while conducting an assessment on a project team trying to improve their processes, I learned that a simple bug had escape through the project’s verification process and resulted in a respin. I refer to this as a simple bug since it could have been easily caught by running a static linting tool. The interesting fact related to this particular bug was that the project team was fairly mature in its verification processes—having adopted constrained-random and coverage-driven verification approaches.
Now, don’t misunderstand what I’m saying—I’m certainly not claiming that you only need static linting to solve all your verification challenges. The same could be said for any single verification solution—after all, verification is an exponential problem. However, there are a few valuable lessons in the simple bug example I just gave. That is, the longer a bug goes undetected, the more expensive it is to fix—and that multiple verification solutions are required to minimize risk for today’s complex designs.
One interesting verification solution that has emerged recently uses formal verification technology to automatically perform a set of design checks for a common set of RTL problems. The key point here is that the user does not need to be an expert with formal technology. This aligns well with my philosophy that any solution that is easy to use and finds bugs as early as possible within a verification flow is worth investigating. What’s interesting about this verification solution is that it can verify a class of problems that cannot be found using static linting tools as well as a class of problems that cannot be found with traditional four-state RTL simulation.
If you are interested in learning more about this verification solution, I would like to invite you to check out the web seminar “Questa Formal’s AutoCheck – The Push-Button Way to Find Bugs,” which is part of our free Transforming Verification On Demand Series of webseminars.
Part 9: The 2010 Wilson Research Group Functional Verification Study
Verification Techniques & Technologies Adoption Trends
This blog is a continuation of a series of blogs, which present the highlights from the 2010 Wilson Research Group Functional Verification Study (for a background on the study, click here).
In my previous blog (Part 8 click here), I focused on some of the 2010 Wilson Research Group findings related to design and verification language trends. In this blog, I present verification techniques and technologies adoption trends, as identified by the 2010 Wilson Research Group study.
One of the claims I made in the prologue to this series of blogs is that we are seeing a trend in increased industry adoption of advanced functional verification techniques, which is supported by the data I present in this blog. An interesting question you might ask is “what is driving this trend?” In some of my earlier blogs (click here for Part 1 and Part 2) I showed an industry trend in that design complexity is increasing in terms design sizes and number of embedded processors. In addition, I’ve presented trend data that showed an increase in total project time and effort spent in verification (click here for Part 4). My belief is that the industry is being forced to mature its functional verification process to address increasing complexity and effort.
Simulation Techniques Adoption Trends
Let’s begin by comparing the adoption trends related to various simulation techniques as shown in Figure 1, where the data from the 2007 Far West Research study is shown in blue and the data from 2010 Wilson Research Group study is shown in green. 
Figure 1. Simulation-based technique adoption trends
You can see that the study finds the industry increasing its adoption of various functional verification techniques.
For example, in 2007, the Far West Research Group found that only 48 percent of the industry performed code coverage. This surprised me. After all, HDL-based code coverage is a technology that has been around since the early 1990’s. However, I did informally verify the 2007 results through numerous customer visits and discussions. In 2010, we see that the industry adoption of code coverage has increased to 72 percent.
Now, a side comment: In this blog, I do not plan to discuss either the strengths or weaknesses of the various verification techniques that were studied (such as code coverage, whose strengths and weaknesses have been argued and debated for years)—perhaps in a separate series of future blogs. In this series of blogs, I plan to focus only on the findings from the 2010 Wilson Research Group study.
In 2007, the Far West Research Group study found that 37 percent of the industry had adopted assertions for use in simulation. In 2010, we find that industry adoption of assertions had increased to 72 percent. I believe that the maturing of the various assertion language standards has contributed to this increased adoption.
In 2007, the Far West Research Group study found that 40 percent of the industry had adopted functional coverage for use in simulation. In 2010, the industry adoption of functional coverage had increased to 69 percent. Part of this increase in functional coverage adoption has been driven by the increased adoption of constrained-random simulation, since you really can’t effectively do constrained-random simulation without doing functional coverage.
In fact, we see from the Far West Research Group 2007 study that 41 percent of the industry had adopted constrained-random simulation techniques. In 2010, the industry adoption had increased to 69 percent. I believe that this increase in constrained-random adoption has been driven by the increased adoption of the various base-class library methodologies, as I presented in a previous blog (click here for Part 8).
Formal Property Checking Adoption Trends
Figure 2 shows the trends in terms of formal property checking adoption by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green). The industry adoption of formal property checking has increased by an amazing 53 percent in the past three years. Again, this is another data point that supports my claim that the industry is starting to mature its adoption of advanced functional verification techniques.
Figure 2. Trends in formal property checking adoption
Another way to analyze the results is to partition a project’s adoption of formal property checking by design size, as shown in Figure 3, where less than 1M gates is shown in blue, 1M to 20M gates is shown in orange, and greater than 20M gates is shown in red. Obviously, the larger the design, the more effort is generally spent in verification. Hence, it’s not too surprising to see the increased adoption of formal property checking for larger designs.
Figure 3. Trends in formal property checking adoption by design size
Acceleration/Emulation & FPGA Prototyping Adoption Trends
The amount of time spent in a simulation regression is an increasing concern for many projects. Intuitively, we tend to think that the design size influences simulation performance. However, there are two equally important factors that must be considered: number of test in the simulation regression suite, and the length of each test in terms of clock cycles.
For example, a project might have a small or moderate-sized design, yet verification of this designs requires a long running test (e.g., a video input stream). Hence, in this example, the simulation regression time is influenced by the number of clock cycles required for the test, and not necessarily the design size itself.
In blog 6 of this series, I presented industry data on the number of tests created to verify a design in simulation (i.e., the regression suite). The findings obviously varied dramatically from a handful of test to thousands of test in a regression suite, depending on the design. In Figure 4 below, I show the findings for a projects regression time, which also varies dramatically from short regression times for some projects, to multiple days for other projects. The median simulation regression time is about 16 hours in 2010. 
Figure 4. Simulation regression time trends
One technique that is often used to speed up simulation regression (either due to very long test and lots of test) is either hardware assisted acceleration or emulation. In addition, FPGA prototyping, while historically used as a platform for software development, has recently served a role in SoC integration validation.
Figure 5 shows the adoption trend for both HW assisted acceleration/emulation and FPGA prototyping by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green). We have seen a 75 percent increase in the adoption of HW assisted acceleration/emulation over the past three years. 
Figure 5. HW assisted acceleration/emulation and FPGA Prototyping trends
I was surprised to see that the adoption of FPGA prototyping did not increase over the past three years, considering that we found in increase in SoC development over the same period. So, I decided to dig deeper into this anomaly.
In Figure 6, you’ll see that I partitioned the HW assisted acceleration/emulation and FPGA prototyping adoption data by design size: less than 1M gates (in blue), 1M to 20M gates (in yellow), and greater than 20M gates (in red). This revealed that the adoption of HW assisted acceleration/emulation continued to increase as design sizes increased. However, the adoption of FPGA prototyping rapidly dropped off as design sizes increased beyond 20M gates. 
Figure 6. Acceleration/emulation & FPGA Prototyping adoption by design size
So, what’s going on? One problem with FPGA prototyping of large designs is that there is an increased engineering effort required to partition designs across multiple FPGAs. In fact, what I have found is that FPGA prototyping of very large designs is often a major engineering effort in itself, and that many projects are seeking alternative solutions to address this problem.
In my next blog, I plan to present the 2010 Wilson Research Group study findings related to various project results in terms of schedule and required spins before production.
Tags: emulation, formal verification, functional verification, Simulation, testbench
Part 8: The 2010 Wilson Research Group Functional Verification Study
Language and Library Trends
This blog is a continuation of a series of blogs, which present the highlights from the 2010 Wilson Research Group Functional Verification Study (for a background on the study, click here).
In my previous blog (Part 7 click here), I focused on some of the 2010 Wilson Research Group findings related to testbench characteristics and simulation strategies. In this blog, I present design and verification language trends, as identified by the Wilson Research Group study.
You might note for some of the language and library data I present, the percentage sums to more than one hundred percent. The reason for this is that some perticipant’s projects use multiple languages and multiple methodologies.
Design Languages
Let’s begin by examining the languages used for design, as shown in Figure 1. Here, we compare the results for languages used to design FPGAs (in grey) with languages used to design non-FPGAs (in green).
Figure 1. Languages used for design
Not too surprising, we see that VHDL is the most popular language used for the design of FPGAs, while Verilog and SystemVerilog are the most popular languages used for the design of non-FPGAs.
Figure 2 shows the trends in terms of languages used for design, by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green), as well as the projected design language adoption trends within the next twelve months (in purple). Note that the design language adoption is declining for most of the languages with the exception of SystemVerilog whose adoption is increasing.
Figure 2. Trends in languages used for design
Verification Languages
Next, let’s look at the languages used for verification (that is, languages used to create simulation testbenches). Figure 3 compares the results between FPGA designs (in grey) and non-FPGA designs (in green). 
Figure 3. Languages used in verification to create simulation testbenches
And again, it’s not too surprising to see that VHDL is the most popular language used to create verification testbenches for FPGAs, while SystemVerilog is the most popular language used to create testbenches for non-FPGAs.
Figure 4 shows the trends in terms of languages used to create simulation testbenches by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green), as well as the projected language adoption trends within the next twelve months (in purple). Note that verification language adoption is declining for most of the languages with the exception of SystemVerilog whose adoption is increasing.
Figure 4. Trends in languages used in verification to create simulation testbenches
Now, let’s look at methodology and class library adoption. Figure 5 shows the future trends in terms of methodology and class library adoption by comparing the 2010 Wilson Research Group study (in green) with the projected adoption trends within the next twelve months (in purple). Previous studies did not include data on methodology and class library adoption, so we are unable to show previous trends.
Figure 5. Methodology and class library future trends
The study indicates that the only methodology adoption projected to grow in the next twelve months are OVM and UVM.
Assertion Languages and Libraries
Finally, let’s examine assertion language and library adoption, as shown in Figure 6. Here, we compare the results for FPGA designs (in grey) and non-FPGA designs (in green).
Figure 6. Assertion language and library adoption
SystemVerilog Assertions (SVA) is the most popular assertion language used for both FPGA and non-FPGA designs.
Figure 7 shows the trends in terms assertion language and library adoption by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green), as well as the projected adoption trends within the next twelve months (in purple). Note that the adoption of most of the assertion languages is declining, with the exception of SVA whose adoption is increasing.
Figure 7. Trends in assertion language and library adoption
In my next blog (click here), I plan to focus on the adoption of various verification technologies and techniques used in the industry, as identified by the 2010 Wilson Research Group study.
Tags: 1076, 1364, 1666, 1800, accellera, Add new tag, Assertion-Based Verification, functional verification, IEEE 1800, OVM, Standards, SystemVerilog, UVM, Verification Methodology, verilog, vhdl, vmm
Part 7: The 2010 Wilson Research Group Functional Verification Study
Testbench Characteristics and Simulation Strategies (Continued)
This blog is a continuation of a series of blogs, which present the highlights from the 2010 Wilson Research Group Functional Verification Study (for a background on the study, click here).
In my previous blog (Part 6 click here), I focused on some of the 2010 Wilson Research Group findings related to testbench characteristics and simulation strategies. In this blog, I continue this discussion, and present additional findings specifically related to the number of tests created by a project, as well as the length of time spent in a simulation regression run for various projects.
Percentage directed tests created by a project
Let’s begin by examining the percentage of directed tests that were created by a project, as shown in Figure 1. Here, we compare the results for FPGA designs (in grey) and non-FPGA designs (in green).
Figure 1. Percentage of directed testing by a project
Obviously, the study results are all over the spectrum, where some projects create more directed tests than others. The study data revealed that FPGA design participants tend to belong to a higher percentage of projects that only do directed tests.
Figure 2 shows the median number of directed tests created on a project by region, where North America (in blue), Europe/Israel (in green), Asia (in green), and India (in red). 
Figure 2. Median percentage of directed testing by a project by region
You can see from the results that India seems to spend less time focused on directed testing compared with other regions, which means that India spends more time with alternative stimulus generation methods (such as, constrained-random, processor-driven, or graph-based techniques).
Let’s look at the percentage of directed testing by design size, for non-FPGA projects. The median results are shown in Figure 3, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 3. Median percentage of directed testing by a project by design size
As design sizes increase, there is less reliance on directed testing.
Percentage of project tests that were random or constrained random
Next, let’s look at the percentage of tests that were random or constrained random across multiple projects. Figure 4 compares the results between FPGA designs (in grey) and non-FPGA designs (in green).
Figure 4. Percentage of random or constrained-random testing by a project
And again, the study results indicate that projects are all over the spectrum in their usage of random or constrained-random stimulation generation. Some projects do more, while other projects do less.
Figure 5 shows the median percentage of random or constrained-random testing by region, where North America (in blue), Europe/Israel (in green), Asia (in green), and India (in red).
Figure 5. Median percentage of random or constrained-random testing by region
You can see that the median percentage of random or constrained-random testing by a project is higher in Indian than other regions of the world.
Let’s look at the percentage of random or constrained-random testing by design size, for non-FPGA projects. The median results are shown in Figure 6, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 6. Median percentage of random or constrained-random testing by design size
Smaller designs tend to do less random or constrained-random testing.
Simulation regression time
Now, let’s look at the time that various projects spend in a simulation regression. Figure 7 compares the simulation regression time between FPGA designs (in grey) and non-FPGA designs (in green) from our recent study.
Figure 7. Time spent in a simulation regression by project
And again, we see that FPGA projects tend to spend less time in a simulation regression run compared to non-FPGA projects.
Figure 8 shows the trends in terms of simulation regression time by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green). There really hasn’t been a significant change in the time spent in a simulation regression within the past three years. You will find that some teams spend days or even weeks in a regression. Yet, the industry median is about 16 hours for both the 2007 and 2010 studies.
Figure 8. Simulation regression time trends
Figure 9 shows the median simulation regression time by region, where North America (in blue), Europe/Israel (in green), Asia (in green), and India (in red).
Figure 9. Median simulation regression time by regions
Finally, Figure 10, shows the median simulation regression time by design size, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 10. Median simulation regression time by design size
Obviously, project teams working on smaller designs spend less time in a simulation regression run compared to project teams working on larger designs.
In my next blog (click here), I’ll focus on design and verification language trends, as identified by the 2010 Wilson Research Group study.
Tags: functional verification, Simulation, testbench, Verification
Part 6: The 2010 Wilson Research Group Functional Verification Study
Testbench Characteristics and Simulation Strategies
This blog is a continuation of a series of blogs that present the highlights from the 2010 Wilson Research Group Functional Verification Study (for background on the study, click here).
In my previous blog (click here), I focused on the controversial topic of effort spent in verification. In this blog, I focus on some of the 2010 Wilson Research Group findings related to testbench characteristics and simulation strategies. Although I am shifting the focus away from verification effort, I believe that the data I present in this blog is related to the discussion of overall effort and really needs to be considered.
Time spent in top-level simulation
Let’s begin by examining the percentage of time a project spends in top-level simulation today. Figure 1 compares the time spent in top-level simulation between FPGA designs (in grey) and non-FPGA designs (in green) from our recent study.
Figure 1. Percentage time spent in top-level simulation
Obviously, the study results show that projects are all over the spectrum, where some projects spend less time in top-level simulation, while others spent much more time.
I decided to partition the data by design size (using the same partition that I’ve presented in previous blogs), and then compare the time spent in top-level verification by design size for Non-FPGA designs. Figure 2 shows the results, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 2. Percentage of time spent in top-level simulation by design size
You can see from the results that, unlike previous analysis where the data was partitioned by design size, it didn’t seem to matter for design sizes between 1M-20M gates and design sizes greater than 20M gates. That is, they seem to spend about the same amount of time in top-level verification.
Time spent in subsystem-level simulation
Next, let’s look at the percentage of time that a project spends in subsystem-level simulation (e.g., clusters, blocks, etc.). Figure 3 compares the time spent in subsystem-level simulation between FPGA designs (in grey) and non-FPGA designs (in green) from our recent study.
Figure 3. Percentage of time spent in subsystem-level simulation
Non-FPGA designs seem to spend more time in subsystem-level simulation when compared to FPGA designs. Again, this is probably not too surprising to anyone familiar with traditional FPGA development. Although, as the size and complexity of FPGA designs continue to increase, I suspect we will see more subsystem-level verification. Unfortunately, we can’t present the trends on FPGA designs, as I mentioned in the background blog for the Wilson Research Group Functional Verification Study. However, future studies should be able to leverage the data we collected in the Wilson Research Group study and present FPGA trends.
Figure 4 shows the Non-FPGA results for the time spent in subsystem-level verification, where the data was partitioned by design size: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red). There doesn’t appear to be any big surprise when viewing the data with this partition.
Figure 4. Percentage of time spent in subsystem-level simulation by design size
Number of tests created to verify the design in simulation
Now let’s look at the percentage of projects in terms of the number of tests they create to verify a design using simulation. Figure 5 compares the number of tests created to verify a design between FPGA designs (in grey) and non-FPGA designs (in green) from our recent study.
Figure 5. Number of tests created to verify a design in simulation
Again, the data is not too surprising. We see that engineers working on FPGA designs tend to create fewer tests to verify their design in simulation.
Figure 6 shows the trend in terms of the number of tests created to verify a design in simulation by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green). You will note that there has been an increase in the number of tests in the last three years. In fact, although there is a wide variation in the number of tests created by a project, the median calculation trend has grown from 342 to 398 tests.
Figure 6. Number of tests created to verify a design in simulation trends
I also did a regional analysis of the number of tests a non-FPGA project creates to verify a design in simulation, and the median results are shown in Figure 7, where North America (in blue), Europe/Israel (in green), Asia (in yellow), and India (in red). You can see that the median calculation is higher in Asia and Indian than North America and Europe/Israel.
Figure 7. Number of tests created to verify a design in simulation by region
Finally, I did a design size analysis of the number of tests a non-FPGA project creates to verify a design in simulation, and the median results are shown in Figure 8, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 8. Number of tests created to verify a design in simulation by design size
Obviously, larger designs require more tests to verify them in simulation.
In my next blog (click here), I’ll continue focusing on testbench characteristics and simulation strategies, and I’ll present additional data from the 2010 Wilson Research Group study.
Tags: functional verification, Simulation, testbench, Verification, Wilson Research Group Study
About Harry Foster
Follow us on Twitter
Harry Foster is Chief Scientist Verification for Mentor Graphics. Prior to EDA, Harry has researched and developed functional verification tools and methodologies for over 12 years as a senior member of the CAD technical staff at Hewlett-Packard, and was an ASIC and board design engineer for 5 years at TI. Harry currently serves as chair of the Accellera Formal Verification Technical Committee, and chair of the IEEE 1850 Property Specification Language (PSL) working group. He is co-author of six books on verification. Harry holds multiple patents in functional verification. He is the original creator of the Accellera Open Verification Library (OVL) assertion monitor standard, and was the 2006 recipient of the Accellera Technical Excellence Award for his contributions to industry standards.
Latest Posts
- Part 1: The 2012 Wilson Research Group Functional Verification Study
- What’s the deal with those wire’s and reg’s in Verilog
- Getting AMP’ed Up on the IEEE Low-Power Standard
- Prologue: The 2012 Wilson Research Group Functional Verification Study
- Even More UVM Debug in Questa 10.2
- IEEE Approves New Low Power Standard
Subscribe to the Verification Horizons Newsletter