Verification Horizons BLOG

This blog will provide an online forum to provide weekly updates on concepts, values, standards, methodologies and examples to assist with the understanding of what advanced functional verification technologies can do and how to most effectively apply them. We're looking forward to your comments and suggestions on the posts to make this a useful tool.

12 December, 2014

Just in time for Christmas and other year-end holidays, I am pleased to announce that the latest issue of Verification Horizons is now available on-line. As usual, we have a great lineup of articles that I’m sure you’ll find informative:

By the way, if you’d like to subscribe to our quarterly Verification Horizons newsletter, you may do so here.

Whichever holiday(s) you celebrate at this special time of year, I hope you experience the peace, joy and hope that my family and I will share this Christmas.

24 November, 2014

SystemVerilog Testbench Debug – Are we having fun yet?


Debug should be fun. Watching waveforms march by, seeing ERRORS and WARNINGS pop out in a transcript file, tracing drivers back to their source, understanding race conditions between simulators and between source code changes – and my favorite – debugging random stability issues. Fun.

Old School – logfiles and interactive

Or at least it should be fun. It used to be fun. I’d setup my collection of scripts to run tests and examine logfiles. Push the button and go for coffee or go home. The next day I’d examine log files and figure out what happened. Usually I’d have to jump into interactive simulation and debug on the fly. Set some breakpoints and watch what happened. That was then. My tests and RTL were all Verilog. Life was good. I was in control of what was going on, and could get my head around it.

New School – logfiles, interactive and class handles

Fast-forward to today. Still have scripts to run tests. Still have log files. Still push the button and get coffee or go home. Still jump into interactive simulation. Still set breakpoints. But now my tests are SystemVerilog class-based – usually UVM. My tests are C code. My tests are constrained random tests. Debug just got harder. I can’t fit the whole testbench + RTL into my head at once. I need help.

Debugging your class based testbench

I prefer to do as much debug as possible in “post-sim” mode. I want to run simulation and capture as much as possible. Then debug my wavefile and source code. What to do about my SystemVerilog class based testbench? Easy. Capture my classes in the wave database. Show them to me in the wave window.

<UVM Testbench class hierarchy window and those same classes in the wave window>

Wave Window

Wave Window

But that’s not possible. Is it? What IS possible?

What? Objects in the wave database? Yes. Objects and their members in the wave database.

Examine the values of class member variables in post-sim mode. Use the waveform window for classes and class member variables just like signals.

What about the handles that are in my classes? Can I chase them to other objects? Yes. Follow class handle “pointers” to other objects – essentially exploring the OBJECT SPACE that existed at THAT time during simulation. But I’m in post sim!

Can I see all the sequence items that hit my driver? Yes. How? Just put the driver “handle” into the wave window and “open” it. You can see the virtual interface handle (if you have one). You can see the transactions that went through the driver (the driver did a ‘get_next_item (t)’ 100,000 times!).

<Transaction handle ‘t’ from the driver in the wave window, with the driver’s virtual interface>

Driver and 't' in Wave Window

Driver and ‘t’ in Wave Window

In the wave window? Yes. All 100,000 of them? Yes.

Now I’m having fun again. That’s great. I can see what’s going on inside my objects. In post-sim mode.

What’s NOT possible?

Will it babysit? No. One thing at a time.

Are you having fun yet?

Find more details in Verification Horizons article – Old School vs. New School – Visualizer and on Verification Academy – Verification and Debug: Old School Meets New School 

You can find all the sessions on New School verification techniques via the following link:

, , , , , ,

17 November, 2014

Few verification tasks are more challenging than trying to achieve code coverage goals for a complex system that, by design, has numerous layers of configuration options and modes of operation.  When the verification effort gets underway and the coverage holes start appearing, even the most creative and thorough UVM testbench architect can be bogged down devising new tests – either constrained-random or even highly directed tests — to reach uncovered areas.

At the recent ARM® Techcon, Nguyen Le, a Principal Design Verification Engineer in the Interactive Entertainment Business Unit at Microsoft Corp. documented a real world case study on this exact situation.  Specifically, in the paper titled “Advanced Verification Management and Coverage Closure Techniques”, Nguyen outlined his initial pain in verification management and improving cover closure metrics, and how he conquered both these challenges – speeding up his regression run time by 3x, while simultaneously moving the overall coverage needle up to 97%, and saving 4 man-months in the process!  Here are the highlights:

* DUT in question
— SoC with multi-million gate internal IP blocks
— Consumer electronics end-market = very high volume production = very high cost of failure!

* Verification flow
— Constrained-random, coverage driven approach using UVM, with IP block-level testbenches as well as  SoC level
— Rigorous testplan requirements tracking, supported by a variety of coverage metrics including functional coverage with SystemVerilog covergroups, assertion coverage with SVA covers, and code coverage on statements, Branches, Expressions, Conditions, and FSMs

* Sign-off requirements
— All test requirements tracked through to completion
— 100% functional, assertion and code coverage

* Pain points
— Code coverage: code coverage holes can come from a variety of expected and unforeseen sources: dead code can be due to unused functions in reused IP blocks, from specific configuration settings, or a bug in the code.  Given the rapid pace of the customer’s development cycle, it’s all too easy for dead code to slip into the DUT due to the frequent changes in the RTL, or due to different interpretations of the spec.  “Unexplainably dead” code coverage areas were manually inspected, and the exclusions for properly unreachable code were manually addressed with the addition of pragmas.  Both procedures were time consuming and error prone
— Verification management: the verification cycle and the generated data were managed through manually-maintained scripting.  Optimizing the results display, throughput, and tool control became a growing maintenance burden.

* New automation
— Questa Verification Manager: built around the Unified Coverage Database (UCDB) standard, the tool supports a dynamic verification plan cross-linked with the functional coverage points and code coverage of the DUT.  In this way the dispersed project teams now had a unified view which told them at a glance which tests were contributing the most value, and which areas of the DUT needed more attention.  In parallel, the included administrative features enabled efficient control of large regressions, merging of results, and quick triage of failures.

— Questa CoverCheck: this tool reads code coverage results from simulation in UCDB, and then leverages formal technology under-the-hood to mathematically prove that no stimulus could ever activate the code in question. If it’s OK for a given block of code to be dead due to a particular configuration choice, etc., the user can automatically generate wavers to refine the code coverage results.  Additionally, the tool can also identify segments of code that, though difficult to reach, might someday be exercised in silicon. In such cases, CoverCheck helps point the way to testbench enhancements to better reach these parts of the design.

— The above tools used in concert (along with Questasim) enabled a very straightforward coverage score improvement process as follows:
1 – Run full regression and merge the UCDB files
2 – Run Questa CoverCheck with the master UCDB created in (1)
3 – Use CoverCheck to generate exclusions for “legitimate” unreachable holes, and apply said exclusions to the UCDB
4 – Use CoverCheck to generate waveforms for reachable holes, and share these with the testbench developer(s) to refine the stimulus
5 – Report the new & improved coverage results in Verification Manager

* Results
— Automation with Verification Manager enabled Microsoft to reduce the variation of test sequences from 10x runtime down to a focused 2x variation.  Additionally, using the coverage reporting to rank and optimize their tests, they increased their regression throughput by 3x!
— With CoverCheck, the Microsoft engineers improved code coverage by 10 – 15% in most hand-coded RTL blocks, saw up to 20% coverage improvement for auto-generated RTL code, and in a matter of hours were able to increase their overall coverage number from 87% to 97%!
— Bottom-line: the customer estimated that they saved 4 man-months on one project with this process

2014 MSFT presentation at ARM Techcon -- cover check ROI

Taking a step back, success stories like this one, where automated, formal-based applications leverage the exhaustive nature of formal analysis to tame once intractable problems (which require no prior knowledge of formal or assertion-based verification), are becoming more common by the day.  In this case, Mentor’s formal-based CoverCheck is clearly the right tool for this specific verification need, literally filling in the gaps in a traditional UVM testbench verification flow.  Hence, I believe the overall moral of the story is a simple rule of thumb: when you are grappling with a “last mile problem” of unearthing all the unexpected, yet potentially damaging corner cases, consider a formal-based application as the best tool for job.  Wouldn’t you agree?

Joe Hupcey III

Reference links:

Direct link to the presentation slides:

ARM Techcon 2014 Proceedings:

Official paper citation:
Advanced Verification Management and Coverage Closure Techniques, Nguyen Le, Microsoft; Harsh Patel, Roger Sabbagh, Darron May, Josef Derner, Mentor Graphics

, , , , , , , , , , ,

5 November, 2014

Between 2006 and 2014, the average number of IPs integrated into an advanced SoC increased from about 30 to over 120. In the same period, the average number of embedded processors found in an advanced SoC increased from one to as many as 20. However, increased design size is only one dimension of the growing verification complexity challenge. Beyond this growing-functionality phenomenon are new layers of requirements that must be verified. Many of these verification requirements did not exist ten years ago, such as multiple asynchronous clock domains, interacting power domains, security domains, and complex HW/SW dependencies. Add all these challenges together, and you have the perfect storm brewing.

It’s not just the challenges in design and verification that have been changing, of course. New technologies have been developed to address emerging verification challenges. For example, new automated ways of applying formal verification have been developed that allow non-Formal experts to take advantage of the significant benefits of formal verification. New technology for stimulus generation have also been developed that allow verification engineers to develop complex stimulus scenarios 10x more efficiently than with directed tests and execute those tests 10x more efficiently than with pure-random generation.

It’s not just technology, of course. Along with new technologies, new methodologies are needed to make adoption of new technologies efficient and repeatable. The UVM is one example of these new methodologies that make it easier to build complex and modular testbench environments by enabling reuse – both of verification components and knowledge.

The Verification Academy website provides great resources for learning about new technologies and methodologies that make verification more effective and efficient. This year, we tried something new and took Verification Academy on the road with live events in Austin, Santa Clara, and Denver. It was great to see so many verification engineers and managers attending to learn about new verification techniques and share their experiences applying these techniques with their colleagues.


If you weren’t able to attend one of the live events – or if you did attend and really want to see a particular session again – you’re in luck. The presentations from the Verification Academy Live seminars are now available on the Verification Academy site:

  • Navigating the Perfect Storm: New School Verification Solutions
  • New School Coverage Closure
  • New School Connectivity Checking
  • New School Stimulus Generation Techniques
  • New School Thinking for Fast and Efficient Verification using EZ-VIP
  • Verification and Debug: Old School Meets New School
  • New Low Power Verification Techniques
  • Establishing a company-wide verification reuse library with UVM
  • Full SoC Emulation from Device Drivers to Peripheral Interfaces

You can find all the sessions via the following link:

, , , , , , , , ,

5 November, 2014

When I was at DAC this year, I had a few folks come up to me at our Verification Academy booth and suggest that I do a “Recipe of the Month” webinar on UVM sequences. They’d seen plenty of information on sequence mechanics and generating stimulus with sequences, but these users were particularly interested in Slave sequences, which aren’t necessarily very intuitive. Of course, we have several articles dealing with all types of sequences in the UVM Cookbook on Verification Academy, but sometimes it’s a little easier to have someone walk through an example in a webinar. To that end, we created an on-demand webinar to explain “UVM Sequences in Depth” which is available for you to review at your leisure.

In the webinar, we explain how a Slave Sequence can be used to implement “responder” functionality in your testbench and how this use model makes it easier to modify the slave functionality as necessary rather than to swap in a new component. We also show how to use virtual sequences to coordinate the stimulus and slave sequences, and then we wrap up with an example of how to handle interrupt sequences.

This particular webinar is, to date, the most popular of our regular webinar series, with over 300 attendees. I was happy to be able to reach so many of you with this information and hope you found it valuable. Feel free to suggest topics for future webinars in the comments section.

Tom Fitzpatrick

9 October, 2014

DVCon India, held in September 2014 in Bangalore, built on the Indian SystemC User Group meeting events and added a Design & Verification track to its popular system-level design (ESL) track that has been popular for many years.  The main stage played host to the keynote presentations, opening ceremonies and best paper and poster awards.

Several DVCon India keynote presentations, which I will go into more depth later touched on emerging use of virtual platforms in system design and the growing impact India has on design verification.  In particular, Mentor’s CEO, Wally Rhines contrasted Wilson Research survey data on design verification from India and the rest of the world.  A strong adoption of SystemVerilog and its popular methodology, the Universal Verification Methodology (UVM) was clear from the survey results Wally shared.

But even beyond SystemVerilog and UVM, the discuss of what could come next anchored the first day of DVCon India discussion on Accellera’s exploration of “portable stimulus.”  Accellera has a group exploring if the industry is ready to start a standards project on this concept.  And the first day when DVCon India attendees were offered an opportunity to learn about this, the multi-company (Mentor Graphics, Breker & CVC) tutorial on the topic was standing room only.

DVCon Europe – The Stage is Set!

A tutorial slot at DVCon Europe will be devoted to the same topic that was popular at DVCon India.  For DVCon Europe attendees, you will find Tutorial T9, “Creating Portable Tests with a Graph-Based Test Specification” will cover this topic.  Technical representatives from Mentor Graphics and Breker will cover aspects of portable stimulus and offer examples of how it can work.  And early application of the technology will be covered by a representative from IBM.  To cover the topic appropriately, we have modified the presenters listed in the official printed program and full details are available online.  The presenters will be, in this order:

  • Holger Horbach, IBM, Germany
  • Frederic Krampac, Breker, France
  • Staffan Berg, Mentor Graphics, Sweden

Please join us for this tutorial and ensuing conversation and discussion.  Verification productivity is a pressing issue and our ability to better control and create stimulus is a step in the direction to address the verification challenges we all face.

One last note, the concept of “portable stimulus” is language agnostic so no matter which language you use for design and verification, the intention is this technology will be able to help.   The tutorial will help you understand how using a graph-based approach enables the highest degree of verification re-use, from IP block to sub-system to full-system level verification. You will see how it supports verification in SystemVerilog, Verilog, VHDL, C, C/C++, assembly, and even other non-traditional base languages. And it also can be extended from simulation to emulation to FPGA prototyping, and even silicon validation.

I look forward to seeing you at DVCon Europe in Munich!  And if you have not yet registered, please do so to secure your seat.

, , , , , , , , , , ,

12 September, 2014

Verification engineers spend lots of time creating tests. In fact, creating enough tests to verify the design functionality consistently tops the list of verification challenges, according to periodic surveys of our customers.


Top Verification Challenges

One challenge in test creation is that verification occurs in multiple environments. For high-level models, verification might be performed in a SystemC environment. Unit-level verification often occurs in a SystemVerilog UVM environment. At the SoC level, the presence of embedded processor cores means embedded software must be part of the verification picture.

Writing specific tests for each environment only exacerbates the “creating enough tests” problem. Besides, different environments offer different levels of automation for test creation. For example, embedded software environments offer no help in creating tests more productively or creating more comprehensive tests – it’s directed tests all the way.

Portable graph-based stimulus brings advanced and highly-productive stimulus modeling techniques to a wide range of verification languages and environments – from automating sequences and virtual sequences in SystemVerilog to creating automated stimulus in VHDL. From verifying C and SystemC designs for high-level synthesis to automating software-driven test scenarios running on embedded processor cores.

If you are attending either DVCon India or DVCon Europe and are interested in how graph-based portable stimulus can boost your test-creation productivity, please consider attending the graph-based portable stimulus tutorial. Links to the respective conference programs are below.

DVCon India Conference Program:

DVCon Europe Conference Program:

11 September, 2014

From those just beginning to study electronic systems design to the practicing engineer, this is the time of the year when those taking their first steps to learn VHDL, Verilog/SystemVerilog join the academic “back to school” crowd and those who are using design & verification languages in practice are honing skills at industry events around the world.

A new academic year has started and the Mentor Higher Education Program (HEP) is well set to help students at more than 1200 colleges and universities secure access to the same commercial tools and technology used by industry.  It is a real win-win when students learn using the same tools they will use after graduating.  Early exposure and use means better skilled and productive engineers for employers.

The functional verification team at Mentor Graphics knows that many students would prefer to have a local copy of ModelSim on their personal computer to do their course work and smaller projects as they learn VHDL or Verilog.  To help facilitate that we make the ModelSim PE Student Edition available for download without charge.  More than 10,000 students use ModelSim PE Student Edition around the world now in addition to our commercial grade tools they can access in their university labs.

For the practicing engineer, the Verification Academy offers an online community of more than 25,000 design and verification engineers that exchange ideas on a wide variety issues across the numerous standards and methodologies.  If you are not a member of the Verification Academy, I recommend you join.  You will also find the Verification Academy at DAC for one-on-one discussions and even more recently Verification Academy Live daylong seminars which came to Austin and which will be in Santa Clara – as of the writing of this blog.  There is still time to register for the Santa Clara event and I invite you to attend.

As design and verification is global, Accellera realized that DVCon should explore the needs of the global design and verification engineer population as well.  For 2014, DVCon Europe and DVCon India were born from an already successful running SystemC User Group events.  These user-led conferences will be held so engineers in these areas can more easily come together to share experiences and knowledge to ultimately become more productive.

Students and practicing engineers alike can benefit from fee-free access to some of the popular IEEE EDA standards.   While I don’t think reading them alone is the ultimate way to educate yourself, they make great companions to daily design and verification activities.  Accellera has worked with the IEEE to place several EDA standards in the IEEE Standards Association’s “Get™” program.  Almost 16,000 copies of the SystemC standard (1666) and just about the same number of SystemVerilog standards (1800) have been downloaded as of the end of August 2014.  Have you download your free copies yet?

The chart below shows the distribution of nearly 45,000 downloads which have occurred since 2010.  Stay tuned for breaking news on some updates to the EDA standards in the Get program.  When updated, they will replace the versions available now.  So if you want to have the current versions and the ones to come out shortly, you better download your copies now.  If the electronic version is not sufficient for you, the IEEE continues to sell printed versions.


From students to practicing engineers, the season of learning has started.  I encourage you to find your right venue or style of learning and connect with others to advance and improve your design and verification productivity.

, , , , , , , , , ,

20 August, 2014

The ever popular Accellera Design & Verification Conference held annually in Silicon Valley is going global.  Accellera System Initiative has expanded many of its SystemC user group events to be more inclusive of other Accellera and IEEE standards.  In doing so, the local organizers of these events have moved to adopt the popular DVCon USA style to organize their events to include this large complement of standards.  If you want to attend, participate or contribute to the events, follow the links as shared below.

Mentor Graphics is excited to participate and sponsor these user-led events with a keynote address, technical paper presentations and educational tutorials.  We look forward to see you in September for DVCon India in Bangalore and in October for DVCon Europe in Munich.

DVCon Europe (14-15 October 2014 | Munich, Germany) will target the application of standardized languages, tools, and methodologies for the design and verification of electronic systems and integrated circuits. The two day event will feature tutorials on the first day and technical paper presentations and poster sessions on the second day.  The DVCon Europe program list the details of the conference.  It is collocated with the annual Forum on Design Languages (FDL), which runs from 14-16 October 2014 in case you want to extend your stay for an extra third day.

At DVCon Europe Mentor Graphics is collaborating with our industry peers and users on a tutorial titled Enabling Energy-Aware System Level Design with UPF-Based System Level Power Models. As power has become one of the major concerns in design equaling those of feature, function and performance, more advances are needed to address system power challenges.  The tutorial will explore the use of IEEE Std. 1801™ (UPF) and how design and verification flows can best use it.

Mentor Graphics will also sponsor a tutorial titled Creating Portable Tests with a Graph-Based Test Specification.  It will cover an overview of a graph-based test description language that raises the level of verification abstraction to address system level challenges. This technology is being used by many successful verification teams around the world today and it is the technology we have committed to help build a new standard upon in Accellera.

DVCon India (25-26 September 2014 | Bangalore, India) is the first year of the transition of the popular Indian SystemC User Group (ISCUG) meeting into an event that expands to cover topics that bring together all the stakeholder involved in design and verification of IP, SoC, ASIC, FPGA and system level solutions.   The event is over two days with common sessions in the morning for keynote addresses.  The attendees will then break into an ESL track and Design & Verification track for focused sessions.

Mentor Graphics will sponsor a tutorial session as well as host the keynote presentation by Mentor Graphics CEO, Dr. Walden C. Rhines.  Dr. Rhines will review recent Wilson Research Group study results on the ongoing convergence of SoC design practices towards a common methodology, independent of specify tools being use. In this keynote, Dr. Rhines identifies the common attributes of SoC methodology that are emerging, and will highlight specific capability enablers for the further optimization of SoC design verification.

Registration for both events is now open and I hope you have time in your calendar to make it there.  Both events will have an exhibition area where you can also catch up on recent updates to our products and discuss what you think should be added next.  The Mentor Graphics team looks forward to meeting you there!

, , , , , , , ,

12 August, 2014

As some of you may have seen, we release a great DAC edition of Verification Horizons back in June. Unfortunately, we were so focused on preparing for a great Verification Academy Booth at DACBoothPic
that I completely forgot to blog about the issue! Well, as the saying goes, “better late than never.”  In particular, I’d like to call your attention to two articles.

The first is “Best Practices for FPGA and ASIC Development” by my Mentor colleagues Josh Rensch and John Boone, in which they take you through the four development phases of an FPGA project. By separating the process into distinct PLAN, EXECUTE, VERIFY and SUPPORT phases, Josh and John help you establish a manageable process that you can apply to your next project. The key is to identify the techniques, or specific activities, that will be used at each phase, and establish a rigorous set of deliverables for each phase. Just like clear interface specs are necessary for two design blocks to be able to talk to each other, clear deliverables at each phase of the process let different groups work together efficiently to create a robust and well-verified design.

The other article I wanted to point out to you is “Merging SystemVerilog Covergroups by Example” by our friend Eldon Nelson, a verification engineer at Micron Technology. This article explores the options available when defining and using covergroups, establishing some important guidelines that will help you get the most useful information when multiple instances of covergroups are merged together. Be sure to check it out.

The other articles in the issue are:

@dennisbrophy Tweets

  • Loading tweets...

@dave_59 Tweets

  • Loading tweets...

@jhupcey Tweets

  • Loading tweets...