Part 5: The 2012 Wilson Research Group Functional Verification Study
Effort Spent in Verification
This blog is a continuation of a series of blogs that present the highlights from the 2012 Wilson Research Group Functional Verification Study (click here). In my previous blog (click here), I focused on design and verification reuse trends. In this blog, I focus on the controversial topic of the amount of effort spent in verification.
Directly asking study participants how much effort they spend in verification will not work. The reason is that it’s hard to find a paper or article on verification that doesn’t start with the phrase: “Seventy percent of a project’s effort is spent in verification…” In other words, the industry is already biased to respond with this effort value. Yet, there are really no creditable references to quantify this value.
I don’t believe that there is a simple answer to the question, “How much effort was spent on verification in your last project?” In fact, I believe that it is necessary to look at multiple data points derived from multiple questions to truly get a sense of effort spent in verification. And that’s what we did in our functional verification study.
Total Project Time Spent in Verification
To try to assess the effort spent in verification, let’s begin by looking at one data point, which is the total project time spent in verification. Figure 1 shows the trends in total percentage of project time spent in verification for non-FPGA designs by comparing the 2007 Far West Research study (in gray), the 2010 Wilson Research Group study (in blue), and the 2012 Wilson Research Group study (in green).
Figure 1. Percentage of total project time spent in verification for Non-FPGA designs
The graph clearly shows that there are some projects that spend a significant percentage of project time in verification (>80%), while other projects spend significantly less time. Notice that in 2007, the average (mean) project time spent in verification was 49 percent, while the average increased to 56 percent in 2010 and remained the same in 2012.
Figure 2 shows the trends in total percentage of project time spent in verification for FPGA designs by comparing the 2010 Wilson Research Group study (in pink) and the 2012 Wilson Research Group study (in red).
Figure 2. Percentage of total project time spent in verification for FPGA designs
You might note that many FPGA projects tend to spend less time in verification than non-FPGA projects. Traditionally, the strategy for FPGA designs has been to get to the lab as soon as possible and debug issues in the lab. In a future blog I’ll show data that indicates this strategy does not necessarily yield good results in terms of meeting project schedule or quality objectives.
Peak Number of Design and Verification Engineers
Next, let’s look at another data point, the average (mean) peak number of engineers involved on a project. Figure 4 compares the growth in recent years for the average peak number of design engineers (in light green) and verification engineers (in dark green) working on a typical non-FPGA project.
Figure 3. Peak number of design vs. verification engineer trends for non-FPGA projects
Note that there has not been a significant increase in design engineers in the past five years, although design sizes have continued to increase at a Moore’s Law rate. This is partially due to increased adoption of internal and external IP (as I discussed in my previous blog) as well as continued productivity improvements due to automation.
However, the mean peak number of verification engineers working on non-FPGA projects has increased by 75% within the last five years. In fact, today we see (on average) a one-to-one ratio for a project’s peak number of design and verification engineers.
Figure 4 provides a different analysis of the data by partitioning the projects by design sizes, and then calculating the mean peak number of verification engineers by project design. The design size partitions are represented as: less than 5M gates, 5M to 20M gates, and greater than 20M gates.
Figure 4. Mean peak number of verification engineer trends by design size for non-FPGA projects
Figure 5 shows the average (mean) peak number of design engineers (in red) and verification engineers (in pink) working on a typical FPGA project.
Figure 5. Peak number of design vs. verification engineer trends for non-FPGA projects
Also, note that the ratio of design engineers versus verification engineers hasn’t changed within the last two years for FPGA projects. Typically, design engineers on FPGA projects are responsible for verification too, and you will find many projects that do not have verification engineers. This trend, however, will likely change as FPGA designs become more complex. We are already seeing this on some very complex FPGA projects today.
In my next blog (click here), I’ll continue the discussion on effort spent in verification as revealed by the 2012 Wilson Research Group Functional Verification Study.
Posted July 15th, 2013, by Harry Foster
- Loading tweets...
- Loading tweets...
- Portable and Productive Test Creation with Graph-Based Stimulus
- Supporting A Season of Learning
- DVCon Goes Global!
- Better Late Than Never: Magical Verification Horizons DAC Edition
- Accellera Approves UVM 1.2
- Getting More Value from your Stimulus Constraints
- The FPGA Verification Window Is Open
- UVM DVCon 2014 Tutorial Video Online
- Mentor Enterprise Verification Platform Debuts
- New Verification Academy ABV Course
- September 2014 (2)
- August 2014 (2)
- July 2014 (1)
- May 2014 (2)
- April 2014 (2)
- March 2014 (2)
- February 2014 (5)
- January 2014 (1)
- November 2013 (2)
- October 2013 (3)
- September 2013 (2)
- August 2013 (4)
- July 2013 (6)
- Part 7: The 2012 Wilson Research Group Functional Verification Study
- Walking in the Desert or Drinking from a Fire Hose?
- Part 6: The 2012 Wilson Research Group Functional Verification Study
- A Short Class on SystemVerilog Classes
- Part 5: The 2012 Wilson Research Group Functional Verification Study
- Part 4: The 2012 Wilson Research Group Functional Verification Study
- June 2013 (2)
- May 2013 (4)
- April 2013 (2)
- March 2013 (2)
- February 2013 (5)
- January 2013 (1)
- December 2012 (1)
- November 2012 (1)
- October 2012 (4)
- September 2012 (1)
- August 2012 (1)
- July 2012 (6)
- June 2012 (1)
- May 2012 (3)
- March 2012 (1)
- February 2012 (6)
- January 2012 (2)
- December 2011 (2)
- November 2011 (2)
- October 2011 (3)
- September 2011 (1)
- July 2011 (3)
- June 2011 (6)
- Intelligent Testbench Automation Delivers 10X to 100X Faster Functional Verification
- Part 9: The 2010 Wilson Research Group Functional Verification Study
- Verification Horizons DAC Issue Now Available Online
- Accellera & OSCI Unite
- The IEEE’s Most Popular EDA Standards
- UVM Register Kit Available for OVM 2.1.2
- May 2011 (2)
- April 2011 (7)
- User-2-User’s Functional Verification Track
- Part 7: The 2010 Wilson Research Group Functional Verification Study
- Part 6: The 2010 Wilson Research Group Functional Verification Study
- SystemC Day 2011 Videos Available Now
- Part 5: The 2010 Wilson Research Group Functional Verification Study
- Part 4: The 2010 Wilson Research Group Functional Verification Study
- Part 3: The 2010 Wilson Research Group Functional Verification Study
- March 2011 (5)
- February 2011 (4)
- January 2011 (1)
- December 2010 (2)
- October 2010 (3)
- September 2010 (4)
- August 2010 (1)
- July 2010 (3)
- June 2010 (9)
- The reports of OVM’s death are greatly exaggerated (with apologies to Mark Twain)
- New Verification Academy Advanced OVM (&UVM) Module
- OVM/UVM @DAC: The Dog That Didn’t Bark
- DAC: Day 1; An Ode to an Old Friend
- UVM: Joint Statement Issued by Mentor, Cadence & Synopsys
- Static Verification
- OVM/UVM at DAC 2010
- DAC Panel: Bridging Pre-Silicon Verification and Post-Silicon Validation
- Accellera’s DAC Breakfast & Panel Discussion
- May 2010 (9)
- Easier UVM Testbench Construction – UVM Sequence Layering
- North American SystemC User Group (NASCUG) Meeting at DAC
- An Extension to UVM: The UVM Container
- UVM Register Package 2.0 Available for Download
- Accellera’s OVM: Omnimodus Verification Methodology
- High-Level Design Validation and Test (HLDVT) 2010
- New OVM Sequence Layering Package – For Easier Tests
- OVM 2.0 Register Package Released
- OVM Extensions for Testbench Reuse
- April 2010 (6)
- SystemC Day Videos from DVCon Available Now
- On Committees and Motivations
- The Final Signatures (the meeting during the meeting)
- UVM Adoption: Go Native-UVM or use OVM Compatibility Kit?
- UVM-EA (Early Adopter) Starter Kit Available for Download
- Accellera Adopts OVM 2.1.1 for its Universal Verification Methodology (UVM)
- March 2010 (4)
- February 2010 (5)
- January 2010 (5)
- December 2009 (15)
- A Cliffhanger ABV Seminar, Jan 19, Santa Clara, CA
- Truth in Labeling: VMM2.0
- IEEE Std. 1800™-2009 (SystemVerilog) Ready for Purchase & Download
- December Verification Horizons Issue Out
- Evolution is a tinkerer
- It Is Better to Give than It Is to Receive
- Zombie Alert! (Can the CEDA DTC “User Voice” Be Heard When They Won’t Let You Listen)
- DVCon is Just Around the Corner
- The “Standards Corner” Becomes a Blog
- I Am Honored to Honor
- IEEE Standards Association Awards Ceremony
- ABV and being from Missouri…
- Time hogs, blogs, and evolving underdogs…
- Full House – and this is no gamble!
- Welcome to the Verification Horizons Blog!
- September 2009 (2)
- July 2009 (1)
- May 2009 (1)