Archive for Harry Foster
Verification Horizons BLOG
Expanding the Verification Academy!
The philosophy behind our Verification Academy is to provide a comprehensive resource for evolving and maturing your functional verification process skills. We believe that each step an organization takes in evolving verification skills should have measurable results and benefits. With that in mind, I am excited to announce two new modules we are adding to the Verification Academy: UVM Express and Advanced UVM.
Figure 1. Screen shot for the Verification Academy UVM Epxress module.
For some verification teams, the hurdle to implement a UVM-based verification environment is simply getting started. To eliminate this hurdle, Mentor Graphics has just introduced UVM Express, which is a suggested systematic way to progressively adopt a UVM methodology. The beauty of this approach is that UVM Express makes getting started easy and intuitive, and provides measurable productivity gains to a broader scope of design projects, regardless of their past experience and existing skills with UVM.
Our new UVM Express module consists of four sessions for evolving UVM capability. It is not necessary for a project to implement all the recommendations from the four sessions to receive measurable benefits. For example, a project might decide to only adopt the recommendations presented in the first session in the UVM Express module, and then at some future point, decide to evolve their skills further by adopting the recommendations from the second session.
In addition to the UVM Express module, we have released an Advanced UVM module. This module consists of 10 sessions and provides over three hours of material. In this module, we build on the concepts originally covered in our Basic UVM Module to take your UVM understanding to the next level. You will learn how to build tests and verification environments, use the factory and configuration database to customize your verification IP, and create reusable stimulus sequences, including those for multi-layer protocols. We will also introduce the UVM Register layer, showing you how to create a register model and how to write and reuse register-level tests.
We are excited to now offer you three UVM modules within the Verification Academy: Basic UVM, Advanced UVM, and UVM Express. And as always, I welcome your feedback and suggestions.
For more information, visit http://www.verificationacademy.com/.
Tags: UVM, Verification Academy
Verification solutions that help reduce bug cost
I think very few engineers would argue with the claim that the longer a bug goes undetected, the more expensive it is to fix. In fact, the general rule-of-thumb is that the cost to fix a bug increases by an order of magnitude as a project progresses from one milestone to the next. Bugs found before simulation obviously have the lowest cost. Bugs found at block or subsystem simulation are generally easier to debug and incur a lower cost to fix versus bugs found at high levels of integration—such as chip- and system-level simulation. Bugs found during post-silicon validation might require a new spin of the chip, while bugs found after product release can be devastating, costing millions of dollars to fix.
Years ago, while conducting an assessment on a project team trying to improve their processes, I learned that a simple bug had escape through the project’s verification process and resulted in a respin. I refer to this as a simple bug since it could have been easily caught by running a static linting tool. The interesting fact related to this particular bug was that the project team was fairly mature in its verification processes—having adopted constrained-random and coverage-driven verification approaches.
Now, don’t misunderstand what I’m saying—I’m certainly not claiming that you only need static linting to solve all your verification challenges. The same could be said for any single verification solution—after all, verification is an exponential problem. However, there are a few valuable lessons in the simple bug example I just gave. That is, the longer a bug goes undetected, the more expensive it is to fix—and that multiple verification solutions are required to minimize risk for today’s complex designs.
One interesting verification solution that has emerged recently uses formal verification technology to automatically perform a set of design checks for a common set of RTL problems. The key point here is that the user does not need to be an expert with formal technology. This aligns well with my philosophy that any solution that is easy to use and finds bugs as early as possible within a verification flow is worth investigating. What’s interesting about this verification solution is that it can verify a class of problems that cannot be found using static linting tools as well as a class of problems that cannot be found with traditional four-state RTL simulation.
If you are interested in learning more about this verification solution, I would like to invite you to check out the web seminar “Questa Formal’s AutoCheck – The Push-Button Way to Find Bugs,” which is part of our free Transforming Verification On Demand Series of webseminars.
Part 9: The 2010 Wilson Research Group Functional Verification Study
Verification Techniques & Technologies Adoption Trends
This blog is a continuation of a series of blogs, which present the highlights from the 2010 Wilson Research Group Functional Verification Study (for a background on the study, click here).
In my previous blog (Part 8 click here), I focused on some of the 2010 Wilson Research Group findings related to design and verification language trends. In this blog, I present verification techniques and technologies adoption trends, as identified by the 2010 Wilson Research Group study.
One of the claims I made in the prologue to this series of blogs is that we are seeing a trend in increased industry adoption of advanced functional verification techniques, which is supported by the data I present in this blog. An interesting question you might ask is “what is driving this trend?” In some of my earlier blogs (click here for Part 1 and Part 2) I showed an industry trend in that design complexity is increasing in terms design sizes and number of embedded processors. In addition, I’ve presented trend data that showed an increase in total project time and effort spent in verification (click here for Part 4). My belief is that the industry is being forced to mature its functional verification process to address increasing complexity and effort.
Simulation Techniques Adoption Trends
Let’s begin by comparing the adoption trends related to various simulation techniques as shown in Figure 1, where the data from the 2007 Far West Research study is shown in blue and the data from 2010 Wilson Research Group study is shown in green. 
Figure 1. Simulation-based technique adoption trends
You can see that the study finds the industry increasing its adoption of various functional verification techniques.
For example, in 2007, the Far West Research Group found that only 48 percent of the industry performed code coverage. This surprised me. After all, HDL-based code coverage is a technology that has been around since the early 1990’s. However, I did informally verify the 2007 results through numerous customer visits and discussions. In 2010, we see that the industry adoption of code coverage has increased to 72 percent.
Now, a side comment: In this blog, I do not plan to discuss either the strengths or weaknesses of the various verification techniques that were studied (such as code coverage, whose strengths and weaknesses have been argued and debated for years)—perhaps in a separate series of future blogs. In this series of blogs, I plan to focus only on the findings from the 2010 Wilson Research Group study.
In 2007, the Far West Research Group study found that 37 percent of the industry had adopted assertions for use in simulation. In 2010, we find that industry adoption of assertions had increased to 72 percent. I believe that the maturing of the various assertion language standards has contributed to this increased adoption.
In 2007, the Far West Research Group study found that 40 percent of the industry had adopted functional coverage for use in simulation. In 2010, the industry adoption of functional coverage had increased to 69 percent. Part of this increase in functional coverage adoption has been driven by the increased adoption of constrained-random simulation, since you really can’t effectively do constrained-random simulation without doing functional coverage.
In fact, we see from the Far West Research Group 2007 study that 41 percent of the industry had adopted constrained-random simulation techniques. In 2010, the industry adoption had increased to 69 percent. I believe that this increase in constrained-random adoption has been driven by the increased adoption of the various base-class library methodologies, as I presented in a previous blog (click here for Part 8).
Formal Property Checking Adoption Trends
Figure 2 shows the trends in terms of formal property checking adoption by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green). The industry adoption of formal property checking has increased by an amazing 53 percent in the past three years. Again, this is another data point that supports my claim that the industry is starting to mature its adoption of advanced functional verification techniques.
Figure 2. Trends in formal property checking adoption
Another way to analyze the results is to partition a project’s adoption of formal property checking by design size, as shown in Figure 3, where less than 1M gates is shown in blue, 1M to 20M gates is shown in orange, and greater than 20M gates is shown in red. Obviously, the larger the design, the more effort is generally spent in verification. Hence, it’s not too surprising to see the increased adoption of formal property checking for larger designs.
Figure 3. Trends in formal property checking adoption by design size
Acceleration/Emulation & FPGA Prototyping Adoption Trends
The amount of time spent in a simulation regression is an increasing concern for many projects. Intuitively, we tend to think that the design size influences simulation performance. However, there are two equally important factors that must be considered: number of test in the simulation regression suite, and the length of each test in terms of clock cycles.
For example, a project might have a small or moderate-sized design, yet verification of this designs requires a long running test (e.g., a video input stream). Hence, in this example, the simulation regression time is influenced by the number of clock cycles required for the test, and not necessarily the design size itself.
In blog 6 of this series, I presented industry data on the number of tests created to verify a design in simulation (i.e., the regression suite). The findings obviously varied dramatically from a handful of test to thousands of test in a regression suite, depending on the design. In Figure 4 below, I show the findings for a projects regression time, which also varies dramatically from short regression times for some projects, to multiple days for other projects. The median simulation regression time is about 16 hours in 2010. 
Figure 4. Simulation regression time trends
One technique that is often used to speed up simulation regression (either due to very long test and lots of test) is either hardware assisted acceleration or emulation. In addition, FPGA prototyping, while historically used as a platform for software development, has recently served a role in SoC integration validation.
Figure 5 shows the adoption trend for both HW assisted acceleration/emulation and FPGA prototyping by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green). We have seen a 75 percent increase in the adoption of HW assisted acceleration/emulation over the past three years. 
Figure 5. HW assisted acceleration/emulation and FPGA Prototyping trends
I was surprised to see that the adoption of FPGA prototyping did not increase over the past three years, considering that we found in increase in SoC development over the same period. So, I decided to dig deeper into this anomaly.
In Figure 6, you’ll see that I partitioned the HW assisted acceleration/emulation and FPGA prototyping adoption data by design size: less than 1M gates (in blue), 1M to 20M gates (in yellow), and greater than 20M gates (in red). This revealed that the adoption of HW assisted acceleration/emulation continued to increase as design sizes increased. However, the adoption of FPGA prototyping rapidly dropped off as design sizes increased beyond 20M gates. 
Figure 6. Acceleration/emulation & FPGA Prototyping adoption by design size
So, what’s going on? One problem with FPGA prototyping of large designs is that there is an increased engineering effort required to partition designs across multiple FPGAs. In fact, what I have found is that FPGA prototyping of very large designs is often a major engineering effort in itself, and that many projects are seeking alternative solutions to address this problem.
In my next blog, I plan to present the 2010 Wilson Research Group study findings related to various project results in terms of schedule and required spins before production.
Tags: emulation, formal verification, functional verification, Simulation, testbench
Part 8: The 2010 Wilson Research Group Functional Verification Study
Language and Library Trends
This blog is a continuation of a series of blogs, which present the highlights from the 2010 Wilson Research Group Functional Verification Study (for a background on the study, click here).
In my previous blog (Part 7 click here), I focused on some of the 2010 Wilson Research Group findings related to testbench characteristics and simulation strategies. In this blog, I present design and verification language trends, as identified by the Wilson Research Group study.
You might note for some of the language and library data I present, the percentage sums to more than one hundred percent. The reason for this is that some perticipant’s projects use multiple languages and multiple methodologies.
Design Languages
Let’s begin by examining the languages used for design, as shown in Figure 1. Here, we compare the results for languages used to design FPGAs (in grey) with languages used to design non-FPGAs (in green).
Figure 1. Languages used for design
Not too surprising, we see that VHDL is the most popular language used for the design of FPGAs, while Verilog and SystemVerilog are the most popular languages used for the design of non-FPGAs.
Figure 2 shows the trends in terms of languages used for design, by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green), as well as the projected design language adoption trends within the next twelve months (in purple). Note that the design language adoption is declining for most of the languages with the exception of SystemVerilog whose adoption is increasing.
Figure 2. Trends in languages used for design
Verification Languages
Next, let’s look at the languages used for verification (that is, languages used to create simulation testbenches). Figure 3 compares the results between FPGA designs (in grey) and non-FPGA designs (in green). 
Figure 3. Languages used in verification to create simulation testbenches
And again, it’s not too surprising to see that VHDL is the most popular language used to create verification testbenches for FPGAs, while SystemVerilog is the most popular language used to create testbenches for non-FPGAs.
Figure 4 shows the trends in terms of languages used to create simulation testbenches by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green), as well as the projected language adoption trends within the next twelve months (in purple). Note that verification language adoption is declining for most of the languages with the exception of SystemVerilog whose adoption is increasing.
Figure 4. Trends in languages used in verification to create simulation testbenches
Now, let’s look at methodology and class library adoption. Figure 5 shows the future trends in terms of methodology and class library adoption by comparing the 2010 Wilson Research Group study (in green) with the projected adoption trends within the next twelve months (in purple). Previous studies did not include data on methodology and class library adoption, so we are unable to show previous trends.
Figure 5. Methodology and class library future trends
The study indicates that the only methodology adoption projected to grow in the next twelve months are OVM and UVM.
Assertion Languages and Libraries
Finally, let’s examine assertion language and library adoption, as shown in Figure 6. Here, we compare the results for FPGA designs (in grey) and non-FPGA designs (in green).
Figure 6. Assertion language and library adoption
SystemVerilog Assertions (SVA) is the most popular assertion language used for both FPGA and non-FPGA designs.
Figure 7 shows the trends in terms assertion language and library adoption by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green), as well as the projected adoption trends within the next twelve months (in purple). Note that the adoption of most of the assertion languages is declining, with the exception of SVA whose adoption is increasing.
Figure 7. Trends in assertion language and library adoption
In my next blog (click here), I plan to focus on the adoption of various verification technologies and techniques used in the industry, as identified by the 2010 Wilson Research Group study.
Tags: 1076, 1364, 1666, 1800, accellera, Add new tag, Assertion-Based Verification, functional verification, IEEE 1800, OVM, Standards, SystemVerilog, UVM, Verification Methodology, verilog, vhdl, vmm
Part 7: The 2010 Wilson Research Group Functional Verification Study
Testbench Characteristics and Simulation Strategies (Continued)
This blog is a continuation of a series of blogs, which present the highlights from the 2010 Wilson Research Group Functional Verification Study (for a background on the study, click here).
In my previous blog (Part 6 click here), I focused on some of the 2010 Wilson Research Group findings related to testbench characteristics and simulation strategies. In this blog, I continue this discussion, and present additional findings specifically related to the number of tests created by a project, as well as the length of time spent in a simulation regression run for various projects.
Percentage directed tests created by a project
Let’s begin by examining the percentage of directed tests that were created by a project, as shown in Figure 1. Here, we compare the results for FPGA designs (in grey) and non-FPGA designs (in green).
Figure 1. Percentage of directed testing by a project
Obviously, the study results are all over the spectrum, where some projects create more directed tests than others. The study data revealed that FPGA design participants tend to belong to a higher percentage of projects that only do directed tests.
Figure 2 shows the median number of directed tests created on a project by region, where North America (in blue), Europe/Israel (in green), Asia (in green), and India (in red). 
Figure 2. Median percentage of directed testing by a project by region
You can see from the results that India seems to spend less time focused on directed testing compared with other regions, which means that India spends more time with alternative stimulus generation methods (such as, constrained-random, processor-driven, or graph-based techniques).
Let’s look at the percentage of directed testing by design size, for non-FPGA projects. The median results are shown in Figure 3, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 3. Median percentage of directed testing by a project by design size
As design sizes increase, there is less reliance on directed testing.
Percentage of project tests that were random or constrained random
Next, let’s look at the percentage of tests that were random or constrained random across multiple projects. Figure 4 compares the results between FPGA designs (in grey) and non-FPGA designs (in green).
Figure 4. Percentage of random or constrained-random testing by a project
And again, the study results indicate that projects are all over the spectrum in their usage of random or constrained-random stimulation generation. Some projects do more, while other projects do less.
Figure 5 shows the median percentage of random or constrained-random testing by region, where North America (in blue), Europe/Israel (in green), Asia (in green), and India (in red).
Figure 5. Median percentage of random or constrained-random testing by region
You can see that the median percentage of random or constrained-random testing by a project is higher in Indian than other regions of the world.
Let’s look at the percentage of random or constrained-random testing by design size, for non-FPGA projects. The median results are shown in Figure 6, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 6. Median percentage of random or constrained-random testing by design size
Smaller designs tend to do less random or constrained-random testing.
Simulation regression time
Now, let’s look at the time that various projects spend in a simulation regression. Figure 7 compares the simulation regression time between FPGA designs (in grey) and non-FPGA designs (in green) from our recent study.
Figure 7. Time spent in a simulation regression by project
And again, we see that FPGA projects tend to spend less time in a simulation regression run compared to non-FPGA projects.
Figure 8 shows the trends in terms of simulation regression time by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green). There really hasn’t been a significant change in the time spent in a simulation regression within the past three years. You will find that some teams spend days or even weeks in a regression. Yet, the industry median is about 16 hours for both the 2007 and 2010 studies.
Figure 8. Simulation regression time trends
Figure 9 shows the median simulation regression time by region, where North America (in blue), Europe/Israel (in green), Asia (in green), and India (in red).
Figure 9. Median simulation regression time by regions
Finally, Figure 10, shows the median simulation regression time by design size, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 10. Median simulation regression time by design size
Obviously, project teams working on smaller designs spend less time in a simulation regression run compared to project teams working on larger designs.
In my next blog (click here), I’ll focus on design and verification language trends, as identified by the 2010 Wilson Research Group study.
Tags: functional verification, Simulation, testbench, Verification
Part 6: The 2010 Wilson Research Group Functional Verification Study
Testbench Characteristics and Simulation Strategies
This blog is a continuation of a series of blogs that present the highlights from the 2010 Wilson Research Group Functional Verification Study (for background on the study, click here).
In my previous blog (click here), I focused on the controversial topic of effort spent in verification. In this blog, I focus on some of the 2010 Wilson Research Group findings related to testbench characteristics and simulation strategies. Although I am shifting the focus away from verification effort, I believe that the data I present in this blog is related to the discussion of overall effort and really needs to be considered.
Time spent in top-level simulation
Let’s begin by examining the percentage of time a project spends in top-level simulation today. Figure 1 compares the time spent in top-level simulation between FPGA designs (in grey) and non-FPGA designs (in green) from our recent study.
Figure 1. Percentage time spent in top-level simulation
Obviously, the study results show that projects are all over the spectrum, where some projects spend less time in top-level simulation, while others spent much more time.
I decided to partition the data by design size (using the same partition that I’ve presented in previous blogs), and then compare the time spent in top-level verification by design size for Non-FPGA designs. Figure 2 shows the results, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 2. Percentage of time spent in top-level simulation by design size
You can see from the results that, unlike previous analysis where the data was partitioned by design size, it didn’t seem to matter for design sizes between 1M-20M gates and design sizes greater than 20M gates. That is, they seem to spend about the same amount of time in top-level verification.
Time spent in subsystem-level simulation
Next, let’s look at the percentage of time that a project spends in subsystem-level simulation (e.g., clusters, blocks, etc.). Figure 3 compares the time spent in subsystem-level simulation between FPGA designs (in grey) and non-FPGA designs (in green) from our recent study.
Figure 3. Percentage of time spent in subsystem-level simulation
Non-FPGA designs seem to spend more time in subsystem-level simulation when compared to FPGA designs. Again, this is probably not too surprising to anyone familiar with traditional FPGA development. Although, as the size and complexity of FPGA designs continue to increase, I suspect we will see more subsystem-level verification. Unfortunately, we can’t present the trends on FPGA designs, as I mentioned in the background blog for the Wilson Research Group Functional Verification Study. However, future studies should be able to leverage the data we collected in the Wilson Research Group study and present FPGA trends.
Figure 4 shows the Non-FPGA results for the time spent in subsystem-level verification, where the data was partitioned by design size: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red). There doesn’t appear to be any big surprise when viewing the data with this partition.
Figure 4. Percentage of time spent in subsystem-level simulation by design size
Number of tests created to verify the design in simulation
Now let’s look at the percentage of projects in terms of the number of tests they create to verify a design using simulation. Figure 5 compares the number of tests created to verify a design between FPGA designs (in grey) and non-FPGA designs (in green) from our recent study.
Figure 5. Number of tests created to verify a design in simulation
Again, the data is not too surprising. We see that engineers working on FPGA designs tend to create fewer tests to verify their design in simulation.
Figure 6 shows the trend in terms of the number of tests created to verify a design in simulation by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green). You will note that there has been an increase in the number of tests in the last three years. In fact, although there is a wide variation in the number of tests created by a project, the median calculation trend has grown from 342 to 398 tests.
Figure 6. Number of tests created to verify a design in simulation trends
I also did a regional analysis of the number of tests a non-FPGA project creates to verify a design in simulation, and the median results are shown in Figure 7, where North America (in blue), Europe/Israel (in green), Asia (in yellow), and India (in red). You can see that the median calculation is higher in Asia and Indian than North America and Europe/Israel.
Figure 7. Number of tests created to verify a design in simulation by region
Finally, I did a design size analysis of the number of tests a non-FPGA project creates to verify a design in simulation, and the median results are shown in Figure 8, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 8. Number of tests created to verify a design in simulation by design size
Obviously, larger designs require more tests to verify them in simulation.
In my next blog (click here), I’ll continue focusing on testbench characteristics and simulation strategies, and I’ll present additional data from the 2010 Wilson Research Group study.
Tags: functional verification, Simulation, testbench, Verification, Wilson Research Group Study
Part 5: The 2010 Wilson Research Group Functional Verification Study
Effort Spent On Verification (Continued)
This blog is a continuation of a series of blogs, which present the highlights from the 2010 Wilson Research Group Functional Verification Study (for a background on the study, click here).
In my previous blog (click here), I focused on the controversial topic of effort spent in verification. This blog continues this discussion.
I stated in my previous blog that I don’t believe there is a simple answer to the question, “how much effort was spent on verification in your last project.” I believe that it is necessary to look at multiple data points to truly get a sense of the real effort involved in verification today. So, let’s look at a few additional findings from the study.
Time designers spend in verification
It’s important to note that verification engineers are not the only project members involved in functional verification. Design engineers spend a significant amount of their time in verification too, as shown in Figure 1.
Figure 1. Mean time designer spends in design vs. verification
In fact, one finding from our study is that the mean time a design engineer spends in verification has increased from an average of 46 percent in 2007, to 50 percent in 2010. The involvement of designers in verification ranges from:
- Small sandbox testing to explore various aspects of the implementation
- Full functional testing of IP blocks and SoC integration
- Debugging verification problems identified by a separate verification team
Percentage of time verification engineers spend on various task
Next, let’s look at the mean time verification engineers spend on various task related to their specific project. You might note that verification engineers spend most of their time in debugging. Ideally, if all the tasks were optimized, then you would expect this. Yet, unfortunately, the time spent in debugging can vary significantly from project-to-project, which presents scheduling challenges for managers during a project’s verification planning process.
Figure 2. Mean time verification engineers spend in different task
Number of formal analysis, FPGA prototyping, and emulation Engineers
Functional verification is not limited only to simulation-based techniques. Hence, it’s important to gather data related to other functional verification techniques, such as the number of verification engineers involved in formal analysis, FPGA prototyping, and emulation.
Figure 3 presents the trends in terms of number of verification engineers focused on formal analysis. In 2007, the median number of verification engineers focused on formal analysis on a project was 1.68, while in 2010 the median number increased to 1.84.
Figure 3. Median number of verification engineers focused on formal analysis
Figure 4 presents the trends in terms of number of verification engineers focused on FPGA prototyping. In 2007, the median number of verification engineers focused on FPGA prototyping on a project was 1.42, while in 2010 the median number increased to 2.04. Although FPGA prototyping is a common technique used to create platforms for software development, it can be used for SoC integration verification and system validation.
Figure 4. Number of verification engineers focused on FPGA prototyping
Figure 5 presents the trends in terms of number of verification engineers focused on hardware assisted acceleration and emulation. In 2007, the median number of verification engineers focused on hardware assisted acceleration and emulation on a project was 1.31, while in 2010 the median number increased to 1.86.
Figure 5. Number of verification engineers focused on emulation
A few more thoughts on verification effort
So, can I conclusively state that 70 percent of a project’s effort is spent in verification today? No. In fact, even after reviewing the data on different aspects of today’s verification process, I would still find it difficult to state quantitatively what the effort is. Yet, the data that I’ve presented so far seems to indicate that the effort (whatever it is) is increasing. And there is still additional data relevant to the verification effort discussion that I plan to present in upcoming blogs. However, in my next blog (click here), I shift the discussion from verification effort, and focus on some of the 2010 Wilson Research Group findings related to testbench characteristics and simulation strategies.
Tags: formal verification, functional verification, Verification
Part 4: The 2010 Wilson Research Group Functional Verification Study
Effort Spent On Verification
This blog is a continuation of a series of blogs, which present the highlights from the 2010 Wilson Research Group Functional Verification Study (click here). In my previous blog (click here), I focused on design and verification reuse trends. In this blog, I focus on the controversial topic of amount of effort spent in verification.
I have been on the technical program committee for many conferences over the past few years (DVCon, DAC, DATE, FDL, HLDVT, MTV . . .), and it seems that there was not a single verification paper that I reviewed that didn’t start with the phrase: “Seventy percent of a project’s effort is spent in verification…blah blah blah.” Yet I’ve always wondered, where did this number come from? There has never been a reliable reference to the origin of this number, and certainly no credible studies that I am aware of.
I don’t believe that there is a simple answer to the question, “how much effort was spent on verification in your last project.” In fact, I believe that it is necessary to look at multiple data points, derived from multiple questions, to truly get a sense of effort spent in verification.
Total Project Time Spent In Verification
To try to assess the effort spent in verification, let’s begin by looking at one data point, which is the total project time spent in verification. Figure 1 shows the trends in total project time spent in verification by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green).
Figure 1. Percentage of total project time spent in verification
Notice that in 2007, the median total project time spent in verification was calculated to be 50 percent, while the number increased to 55 percent in 2010. Our recent study seems to indicate that the time spent in verification is increasing.
Peak Number of Verification Engineers
Next, let’s look at another data point, the peak number of verification engineers on a project. Figure 2 compares the peak number of verification engineers involved on FPGA designs (in grey) and non-FPGA designs (in green) from our recent study.
Figure 2. Peak number of verification engineers
It’s not surprising that projects involving non-FPGA designs tend to have a higher number of peak verification engineers compared with FPGA designs.
Figure 3 shows the trends in peak number of verification engineers for non-FPGA designs by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green).
Figure 3. Peak number of verification engineer trends
I decided that another interesting way to look at the data is to partition the set regionally, and calculate the median peak number of verification engineers on a project by region. The results are shown in Figure 4, with North America (in blue), Europe/Israel (in green), Asia (in green), and India (in red).
Figure 4. Peak number of verification engineer by region
Noticed how, on average, India seems to have more peak engineers involved on a project. India certainly has developed a core set of expertise in verification over the past few year’s.
The next analysis I decided to perform was to partition the data by design size, and then compare the median peak number of verification engineers. Figure 5 shows the results, where the design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 5. Peak number of verification engineer by design size
Although I am focusing on effort spent in verification at the moment, let’s take a look at the peak number of design engineers involved on a project today. Figure 6 compares the peak number of design engineers involved on FPGA designs (in grey) and non-FPGA designs (in green).
Figure 6. Peak number of design engineers
Next, in Figure 7 I show the trends in peak number of design engineers for non-FPGA designs by comparing the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green).
Figure 7. Peak number of design engineer trends
You might note that there has not been a significant increase in design engineers in the past three years, although design sizes have increased. This is partially due to increased adoption of internal and external IP (as I discussed in my previous blog), as well as continued productivity improvements due to automation.
After I saw this data, I thought it would be interesting to compare the median increase in verification engineers to the median increase in design engineers from 2007 to 2010. The results were shocking, as shown in Figure 8, where we see a four percent increase in peak number of design engineers in the last three years compared to a 58 percent increase in peak number of verification engineers. Clearly, verification productivity improvements are needed in the industry to address this problem.
Figure 8. Peak number of design vs. verification engineer trends
In my next blog (click here), I’ll continue the discussion on effort spent in verification as revealed by the 2010 Wilson Research Group Functional Verification Study.
Part 3: The 2010 Wilson Research Group Functional Verification Study
Reuse Trends
This blog is a continuation of a series of blogs, which presents the highlights from the 2010 Wilson Research Group Functional Verification Study (click here). In my previous blog (click here), I focused on embedded processors, power management, and clock domains. In this blog, I focus on design and verification reuse trends. As I mentioned in my prologue blog to this series (click here), one interesting trend that emerged from the study is that reuse adoption is increasing.
Design Composition Trends
Figure 1 shows the median design composition trends, which compares the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green).
Notice that new logic development has decreased by 34 percent in the last three years, while external IP adoption has increased by 69 percent. This increase in adoption has been driven by IP demand required for SoC development, such as embedded processor cores (e.g., ARM cores) and standard interface cores (e.g., USB cores).
Figure 1. Median design composition trends
Verification Testbench Composition Trends
Figure 2 shows the median testbench composition trends, which compares the 2007 Far West Research study (in blue) with the 2010 Wilson Research Group study (in green).
Notice that new verification code development has decreased by 24 percent in the last three years, while external verification IP adoption has increased by 138 percent. This increase has been driven by the emergence of standard on-chip and off-chip bus architectures.
Figure 2. Median testbench composition trends
In my next blog (click here), I’ll shift my focus from design trends to project resource trends. I’ll also present our findings on the project effort spent in verification.
Part 2: The 2010 Wilson Research Group Functional Verification Study
Design Trends (Continued)
In Part 1 of this series of blogs, I focused on design trends (click here) as identified by the 2010 Wilson Research Group Functional Verification Study (click here). In this blog, I continue presenting the study findings related to design trends, with a focus on embedded processor, power management, and clock domain trends.
Embedded Processors
In Figure 1, we see the percentage of today’s designs by the number of embedded processor cores. It’s interesting to note that 78 percent of all non-FPGA designs (as shown in green) contain one or more embedded processors and could be classified as an SoC, which by nature is complex to verify. Yet, even 55 percent of all FPGA designs contain one or more embedded processors.
Figure 1. Number of embedded processor cores
Figure 2 shows the trends in terms of number of embedded processor cores for non-FPGA designs. The comparison includes the 2004 Collett study (in orange), the 2007 Far West Research study (in blue), and the 2010 Wilson Research Group study (in green).
We are unable to show the FPGA trend data since none of the prior industry studies contained FPGA participates. However, future studies should be able to show FPGA trends since the 2010 Wilson Research Group study did contain FPGA participants.
Figure 2. Number of embedded processor core trends
The median number of embedded processor cores in 2004 was about 1.06. This number increased in 2007 to 1.46. Today, the median number of embedded processor cores was found to be 2.14.
Another interesting analysis on the study data is to partition it into design sizes (for example, less than 1M gates, 1M to 20M gates, greater than 20M gates), and then calculate the median number of embedded processors per partitioned set. The results are shown in Figure 3, and as you would expect, the larger the design, the more embedded processor cores.
Figure 3. Median embedded processor cores by design size
Platform-based SoC design approaches, containing multiple embedded processor cores with lots of third-party and internally developed IP, have driven the demand for common bus architectures. In Figure 4 we see the percentage of today’s designs by the type of on-chip bus architecture for both FPGA (in grey) and non-FPGA (in green) designs.
Figure 4. On-chip bus architecture adoption
Figure 5 shows the trends in terms of on-chip bus architecture adoption. The comparison includes the 2007 Far West Research study (in blue), and the 2010 Wilson Research Group study (in green). The study did not partition out the various ARM AMBA bus architectures between the 2007 and 2010 studies. However, it is interesting to note that there was about a two hundred and forty one percent increase in designs using the ARM AMBA bus architecture.
Figure 5. On-chip bus architecture adoption trends
One interesting way to analyze the study data is to partition the responses by geographical region. The results are shown in Figure 6. The regional comparison are North America (in blue), Europe/Israel (in green), Asia minus India (in orange), and India (in red).
Notice how Asia appears to lead the world in the development of designs containing ARM processors when compared to the rest of the world.
Figure 6. On-chip bus architecture adoption by region
Another interesting analysis is to partition the data by design sizes. The results are shown in Figure 7 with the following design size partitions: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 7. On-chip bus architecture adoption by design size
In Figure 8 we see the percentage of today’s designs by the number of embedded DSP cores for both FPGA designs (in grey) and non-FPGA designs (in green).
Figure 8. Number of embedded DSP cores
Figure 9 shows the trends in terms of the number of embedded DSP cores for non-FPGA designs. The comparison includes the 2007 Far West Research study (in blue), and the 2010 Wilson Research Group study (in green).
Figure 9. Number of embedded DSP core trends
Independent Asynchronous Clock Domains
Figure 10 shows the percentage of design developed today by the number of independent asynchronous clock domains. The asynchronous clock domain data for FPGA designs are shown in grey, while the data for the non-FPGA designs is shown in green.
Figure 10. Number of independent asynchronous clock domains
Figure 11 shows the trends in number of independent asynchronous clock domains for non-FPGA designs. The comparison includes the 2002 Collett study (in orange), the 2004 Collett study (in pink), the 2007 Far West Research study (in blue), and the 2010 Wilson Research Group study (in green).
It’s interesting to note that, although the number of clock domains is increasing over time, the sweet spot in terms of number of independent asynchronous clock domains seems to remain between two and 11 clock domains, and hasn’t changed significantly in the past nine years.
Figure 11. Number of independent asynchronous clock domain trends
Figure 12 partitions the study data by geographical region, and shows the median calculation for the number of independent asynchronous clock domains. The regional comparison are North America (in blue), Europe/Israel (in green), Asia mimus India (in orange), and India (in red).
Notice how Asia appears to lead the world in the median number of independent asynchronous clock domains.
Figure 12. Median number of independent clock domains by regions
Figure 13 provides a different analysis of the data by partitioning the data into design sizes, and then calculating the median number of independent asynchronous clock domains. The design size partitions are represented as: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 13. Median number of independent clock domains by design size
Power Management
Figure 14 shows the percentage of designs that actively manage power by process geometry size. You will note that at 45nm, the study indicates that there is an increasing need to actively manage power.
Figure 14. Designs that actively manage power by process geometry
The size of the design, regardless of its process geometry, influences the decision to actively manage power, as shown Figure 15. The design size partitions are represented as follows: less than 1M gates (in blue), 1M to 20M gates (in orange), and greater than 20M gates (in red).
Figure 15. Design that actively manage power by size
Although there are many techniques that are used to manage power, Figure 16 shows the percentage of use for the top eight techniques that were identified through the study. It’s important to note that many designs will implement multiple power management solutions on a single chip.
Figure 16. Top eight techniques used to actively manage power
In my next blog (click here), I’ll present data on design and verification reuse trends.
About Harry Foster
Follow us on Twitter
Harry Foster is Chief Scientist Verification for Mentor Graphics. Prior to EDA, Harry has researched and developed functional verification tools and methodologies for over 12 years as a senior member of the CAD technical staff at Hewlett-Packard, and was an ASIC and board design engineer for 5 years at TI. Harry currently serves as chair of the Accellera Formal Verification Technical Committee, and chair of the IEEE 1850 Property Specification Language (PSL) working group. He is co-author of six books on verification. Harry holds multiple patents in functional verification. He is the original creator of the Accellera Open Verification Library (OVL) assertion monitor standard, and was the 2006 recipient of the Accellera Technical Excellence Award for his contributions to industry standards.
@dennisbrophy tweets
- #IEEE Introduces Groundbreaking Standard for Body Area Networking #in http://t.co/D3Q5d6Xu #
- Meet this #49DAC at the Verification Acadmey booth: http://t.co/iqDbPnkS http://t.co/sgmwQQgv #in #
@dave_59 tweets
- Just posted a 7.60 mi bike - I biked to work on Bike to Work Day. http://t.co/iWpcr9Xu #RunKeeper #
- RT @GordonMcGregor: Simplified SystemVerilog VPI iterators http://t.co/5QCRDhek #verilab #
Latest Posts
- Dave Rich Featured on EEWeb
- How Did I Get Here?
- Expanding the Verification Academy!
- Get on the Fast Track to Advanced Verification with UVM Express
- Introducing UVM Connect
- Tornado Alert!!!
Subscribe to the Verification Horizons Newsletter