Intelligent Testbench Automation Delivers 10X to 100X Faster Functional Verification

iTBA Introduction

If you’ve been to DAC or DVCon during the past couple of years, you’ve probably at least heard of something new called “Intelligent Testbench Automation”.  Well, it’s actually not really all that new, as the underlying principles have been used in compiler testing and some types of software testing for the past three decades, but its application to electronic design verification is certainly new, and exciting.

The value proposition of iTBA is fairly simple and straightforward.  Just like constrained random testing, iTBA generates tons of stimuli for functional verification.  But iTBA is so efficient, that it achieves the targeted functional coverage one to two orders of magnitude faster than CRT.  So what would you do if you could achieve your current simulation goals 10X to 100X faster?

You could finish your verification earlier, especially when it seems like you’re getting new IP drops every day.  I’ve seen IP verification teams reduce their simulations from several days on several CPUs (using CRT) to a couple of hours on a single CPU (with iTBA).  No longer can IP designers send RTL revisions faster than we can verify them.

But for me, I’d ultimately use the time savings to expand my testing goals.  Today’s designs are so complex that typically only a fraction of their functionality gets tested anyway.  And one of the biggest challenges is trading off what functionality to test, and what not to test.  (We’ll show you how iTBA can help you here, in a future blog post.)  Well, if I can achieve my initial target coverage in one-tenth of the time, then I’d use at least part of the time saving to expand my coverage, and go after some of the functionality that originally I didn’t think I’d have time to test.

On Line Illustration

If you check out this link – http://www.verificationacademy.com/infact  – you’ll find an interactive example of a side by side comparison of constrained random testing and intelligent testbench automation.  It’s an Adobe Flash Demonstration, and it lets you run your own simulations.  Try it, it’s fun.

The example shows a target coverage of 576 equally weighted test cases in a 24×24 grid.  You can adjust the dials at the top for the number and speed of simulators to use, and then click on “start”.  Both CRT and iTBA simulations run in parallel at the same speed, cycle for cycle, and each time a new test case is simulated the number in its cell is incremented by one, and the color of the cell changes.  Notice that the iTBA simulation on the right achieves 100% coverage very quickly, covering every unique test case efficiently.  But notice that the CRT simulation on the left eventually achieves 100% coverage painfully and slowly, with much unwanted redundancy.  You can also click on “show chart” to see a coverage chart of your simulation.

Math Facts

You probably knew that random testing repeats, but you probably didn’t know by how much.  It turns out that the redundancy factor is expressed in the equation “ T = N ln N + C “, where “T” is the number of tests that must be generated to achieve 100% coverage of “N” different cases, and “C” is a small constant.  So using the natural logarithm of 576, we can calculate that given equally weighted cases, the random simulation will require an average of about 3661 tests to achieve our goal.  Sometimes it’s more, sometimes it’s less, given the unpredictability of random testing.  In the meantime the iTBA simulation achieves 100% coverage in just 576 tests, a reduction of 84%.

Experiment at Home

You probably already have an excellent six-sided demonstration vehicle somewhere at home.  Try rolling a single die repeatedly, simulating a random test generator.  How many times does it take you to “cover” all six unique test cases?  T = N ln N + C says it should take about 11 times or more.  You might get lucky and hit 8, 9, or 10.  But chances are you’ll still be rolling at 11, 12, 13, or even more.  If you used iTBA to generate the test cases, it would take you six rolls, and you’d be done.  Now in this example, getting to coverage twice as fast may not be that exciting to you.  But if you extrapolate these results to your RTL design’s test plan, the savings can become quite interesting.

Quiz Question

So here’s a quick question for you.  What’s the minimum number of unique functional test cases needed to realize at least a 10X gain in efficiency with iTBA compared to what you could get with CRT?  (Hint – You can figure it out with three taps on a scientific calculator.)  It’s probably a pretty small number compared to the number of functions your design can actually perform, meaning that there’s at least a 10X improvement in testing efficiency awaiting you with iTBA.

More Information

Hopefully at this point you’re at least a little bit interested?  Like some others, you may be skeptical at this point.  Could this technology really offer a 10X improvement in functional verification?  Check out the Verification Academy at this site – http://www.verificationacademy.com/course-modules/dynamic-verification/intelligent-testbench-automation – to see the first academy sessions that will introduce you to Intelligent Testbench Automation.  Or you can even Google “intelligent testbench automation”, and see what you find.  Thanks for reading . . .

Post Author

Posted June 28th, 2011, by

Post Tags

, , , , , ,

Post Comments

3 Comments

About Verification Horizons BLOG

This blog will provide an online forum to provide weekly updates on concepts, values, standards, methodologies and examples to assist with the understanding of what advanced functional verification technologies can do and how to most effectively apply them. We're looking forward to your comments and suggestions on the posts to make this a useful tool. Verification Horizons BLOG

@dennisbrophy Tweets

  • Loading tweets...

@dave_59 Tweets

  • Loading tweets...

@jhupcey Tweets

  • Loading tweets...

Comments

3 comments on this post | ↓ Add Your Own

Commented on July 6, 2011 at 10:33 am
By Blog Review: July 6 | System-Level Design

[…] Mark Olen digs into intelligent testbench automation, which is definitely a step in the right direction for […]

[…] to generate stimulus rather than relying on the methods built into OVM & UVM in his recent blog post.  Mark shows how a user gains 10x-100x  in efficiency all the while doing this from within their […]

Commented on February 21, 2012 at 4:56 pm
By Tornado Alert!!! « Verification Horizons BLOG

[…] My colleague, Mark Olen, blogs about iTBA here.   If you have not had a chance yet to read his blog on iTBA delivering 10x to 100x faster functional verification, it is worth the time to do so.  You can […]

Add Your Comment

Archives

November 2014
  • SystemVerilog Testbench Debug – Are we having fun yet?
  • ARM® Techcon Paper Report: How Microsoft Saved 4 Man-Months Meeting Their Coverage Closure Goals Using Automated Verification Management & Formal Apps
  • Preparing for the Perfect Storm with New-School Verification Techniques
  • On-Demand Webinar: UVM Sequences in Depth
  • October 2014
  • DVCon India: A Smashing Hit!
  • September 2014
  • Portable and Productive Test Creation with Graph-Based Stimulus
  • Supporting A Season of Learning
  • August 2014
  • DVCon Goes Global!
  • Better Late Than Never: Magical Verification Horizons DAC Edition
  • July 2014
  • Accellera Approves UVM 1.2
  • May 2014
  • Getting More Value from your Stimulus Constraints
  • The FPGA Verification Window Is Open
  • April 2014
  • UVM DVCon 2014 Tutorial Video Online
  • Mentor Enterprise Verification Platform Debuts
  • March 2014
  • New Verification Academy ABV Course
  • DVCon 2014 Issue of Verification Horizons Now Available
  • February 2014
  • DVCon–The FREE Side
  • More DVCon–More Mentor Tutorials!
  • UVM 1.2: Open Public Review
  • DVCon 2014: Standards on Display
  • Just because FPGAs are programmable doesn’t mean verification is dead
  • January 2014
  • Managing Verification Coverage Information
  • November 2013
  • Epilogue: The 2012 Wilson Research Group Functional Verification Study
  • New Verification Horizons Issue Available
  • October 2013
  • Happy Halloween from ARM TechCon
  • IEEE Standards Association Symposium on EDA Interoperability
  • STMicroelectronics: Simulation + Emulation = Verification Success
  • September 2013
  • A Decade of SystemVerilog: Unifying Design and Verification?
  • Part 12: The 2012 Wilson Research Group Functional Verification Study
  • August 2013
  • Part 11: The 2012 Wilson Research Group Functional Verification Study
  • Part 10: The 2012 Wilson Research Group Functional Verification Study
  • Part 9: The 2012 Wilson Research Group Functional Verification Study
  • Part 8: The 2012 Wilson Research Group Functional Verification Study
  • July 2013
  • Part 7: The 2012 Wilson Research Group Functional Verification Study
  • Walking in the Desert or Drinking from a Fire Hose?
  • Part 6: The 2012 Wilson Research Group Functional Verification Study
  • A Short Class on SystemVerilog Classes
  • Part 5: The 2012 Wilson Research Group Functional Verification Study
  • Part 4: The 2012 Wilson Research Group Functional Verification Study
  • June 2013
  • Part 3: The 2012 Wilson Research Group Functional Verification Study
  • Part 2: The 2012 Wilson Research Group Functional Verification Study
  • May 2013
  • Texas-Sized DAC Edition of Verification Horizons Now Up on Verification Academy
  • IEEE 1801™-2013 UPF Standard Is Published
  • Part 1: The 2012 Wilson Research Group Functional Verification Study
  • What’s the deal with those wire’s and reg’s in Verilog
  • April 2013
  • Getting AMP’ed Up on the IEEE Low-Power Standard
  • Prologue: The 2012 Wilson Research Group Functional Verification Study
  • March 2013
  • Even More UVM Debug in Questa 10.2
  • IEEE Approves New Low Power Standard
  • February 2013
  • Verification Horizons DVCon Issue Now Available
  • Get your IEEE 1800-2012 SystemVerilog LRM at no charge
  • IEEE 1800™-2012 SystemVerilog Standard Is Published
  • See You at DVCon 2013!
  • Get Ready for SystemVerilog 2012
  • January 2013
  • VHDL Update Comes to Verification Academy!
  • December 2012
  • IEEE Approves Revised SystemVerilog Standard
  • November 2012
  • Coverage Cookbook Debuts
  • October 2012
  • IoT: Internet of Things
  • Check out the October, 2012 Verification Horizons
  • Improving simulation results with formal-based technology
  • Introducing “Verification Academy 2.0”
  • September 2012
  • OVM Gets Connected
  • August 2012
  • OpenStand & EDA Standardization
  • July 2012
  • Synthesizing Hardware Assertions and Post-Silicon Debug
  • Virtual Emulation for Debugging
  • Verification Academy: Up Close & Personal
  • SystemC Standardization Cycle Completes
  • Verification Standards Take Another Step Forward
  • New UVM Recipe of the Month: Scoreboarding in UVM
  • June 2012
  • Intelligent Testbench Automation – Catching on Fast
  • May 2012
  • Two Articles You Need to Check Out
  • Off to DAC!
  • Dave Rich Featured on EEWeb
  • March 2012
  • How Did I Get Here?
  • February 2012
  • Expanding the Verification Academy!
  • Get on the Fast Track to Advanced Verification with UVM Express
  • Introducing UVM Connect
  • Tornado Alert!!!
  • UVM: Some Thoughts Before DVCon
  • UVM™ at DVCon 2012
  • January 2012
  • SystemC 2011 Standard Published
  • Verification solutions that help reduce bug cost
  • December 2011
  • Instant Replay for Debugging SoC Level Simulations
  • 2011 IEEE Design Automation Standards Awards
  • November 2011
  • Getting started with the UVM – Using the Register Modeling package
  • TLM Becomes an IEEE Standard
  • October 2011
  • Worlds Standards Day 2011
  • VHS or Betamax?
  • Verification Issues Take Center Stage
  • September 2011
  • New UVM Recipe-of-the-Month: Sequence Layering
  • July 2011
  • Combining Intelligent Testbench Automation with Constrained Random Testing
  • Going from “Standards Development” to “Standards Practice”
  • Verification Academy Now Includes OVMWorld Content
  • June 2011
  • Intelligent Testbench Automation Delivers 10X to 100X Faster Functional Verification
  • Part 9: The 2010 Wilson Research Group Functional Verification Study
  • Verification Horizons DAC Issue Now Available Online
  • Accellera & OSCI Unite
  • The IEEE’s Most Popular EDA Standards
  • UVM Register Kit Available for OVM 2.1.2
  • May 2011
  • Part 8: The 2010 Wilson Research Group Functional Verification Study
  • Getting Your Standards Update @ DAC 2011
  • April 2011
  • User-2-User’s Functional Verification Track
  • Part 7: The 2010 Wilson Research Group Functional Verification Study
  • Part 6: The 2010 Wilson Research Group Functional Verification Study
  • SystemC Day 2011 Videos Available Now
  • Part 5: The 2010 Wilson Research Group Functional Verification Study
  • Part 4: The 2010 Wilson Research Group Functional Verification Study
  • Part 3: The 2010 Wilson Research Group Functional Verification Study
  • March 2011
  • Part 2: The 2010 Wilson Research Group Functional Verification Study
  • Part 1: The 2010 Wilson Research Group Functional Verification Study
  • Prologue: The 2010 Wilson Research Group Functional Verification Study
  • Language Transitions: The Dawning of Age of Aquarius
  • Using the UVM libraries with Questa
  • February 2011
  • DVCon: The Present and the Future
  • Free at Last! UVM1.0 is Here!
  • Parameterized Classes, Static Members and the Factory Macros
  • IEEE Standards in India
  • January 2011
  • Accellera Approves New Co-Emulation Standard
  • December 2010
  • New Verification Horizons: Methodologies Don’t Have to be Scary
  • The Survey Says: Verification Planning
  • October 2010
  • Towards UVM Register Package Interoperability
  • IEC’s 47th General Assembly Meeting Opens
  • UVM: Giving Users What They Want
  • September 2010
  • UVM Takes Shape in the Accellera VIP-TSC
  • Accellera VIP-TSC Selects RAL for UVM 1.0 Register Package
  • OVM Cookbook Available from OVMWorld.org
  • UVM Register Package Candidate News
  • August 2010
  • Redefining Verification Performance (Part 2)
  • July 2010
  • Making formal property checking easy to use
  • Redefining Verification Performance (Part 1)
  • SystemVerilog Coding Guidelines: Package import versus `include
  • June 2010
  • The reports of OVM’s death are greatly exaggerated (with apologies to Mark Twain)
  • New Verification Academy Advanced OVM (&UVM) Module
  • OVM/UVM @DAC: The Dog That Didn’t Bark
  • DAC: Day 1; An Ode to an Old Friend
  • UVM: Joint Statement Issued by Mentor, Cadence & Synopsys
  • Static Verification
  • OVM/UVM at DAC 2010
  • DAC Panel: Bridging Pre-Silicon Verification and Post-Silicon Validation
  • Accellera’s DAC Breakfast & Panel Discussion
  • May 2010
  • Easier UVM Testbench Construction – UVM Sequence Layering
  • North American SystemC User Group (NASCUG) Meeting at DAC
  • An Extension to UVM: The UVM Container
  • UVM Register Package 2.0 Available for Download
  • Accellera’s OVM: Omnimodus Verification Methodology
  • High-Level Design Validation and Test (HLDVT) 2010
  • New OVM Sequence Layering Package – For Easier Tests
  • OVM 2.0 Register Package Released
  • OVM Extensions for Testbench Reuse
  • April 2010
  • SystemC Day Videos from DVCon Available Now
  • On Committees and Motivations
  • The Final Signatures (the meeting during the meeting)
  • UVM Adoption: Go Native-UVM or use OVM Compatibility Kit?
  • UVM-EA (Early Adopter) Starter Kit Available for Download
  • Accellera Adopts OVM 2.1.1 for its Universal Verification Methodology (UVM)
  • March 2010
  • The Art of Deprecation
  • OVM 2.1.1 Now Ready for Download
  • February 2010 Verification Horizons Newsletter Now Available
  • IEEE Standards Meetings in India
  • February 2010
  • I Do It …
  • SystemVerilog: A time for change? Maybe not.
  • Partners Offer Support for OVM 1.0 Register Package
  • SystemC Day at DVCon
  • OVM/VMM Interoperability Kit: It’s Ready!
  • January 2010
  • Three Perfect 10’s
  • OVM 1.0 Register Package Released
  • Accellera Adopts OVM
  • SystemC (IEEE Std. 1666™) Comes to YouTube
  • Debugging requires a multifaceted solution
  • December 2009
  • A Cliffhanger ABV Seminar, Jan 19, Santa Clara, CA
  • Truth in Labeling: VMM2.0
  • IEEE Std. 1800™-2009 (SystemVerilog) Ready for Purchase & Download
  • December Verification Horizons Issue Out
  • Evolution is a tinkerer
  • It Is Better to Give than It Is to Receive
  • Zombie Alert! (Can the CEDA DTC “User Voice” Be Heard When They Won’t Let You Listen)
  • DVCon is Just Around the Corner
  • The “Standards Corner” Becomes a Blog
  • I Am Honored to Honor
  • IEEE Standards Association Awards Ceremony
  • ABV and being from Missouri…
  • Time hogs, blogs, and evolving underdogs…
  • Full House – and this is no gamble!
  • Welcome to the Verification Horizons Blog!
  • September 2009
  • SystemVerilog: The finer details of $unit versus $root.
  • SystemVerilog Coding Guidelines
  • July 2009
  • The Language versus The Methodology
  • May 2009
  • Are Program Blocks Necessary?