Posts Tagged ‘ARM Techcon’

17 November, 2014

Few verification tasks are more challenging than trying to achieve code coverage goals for a complex system that, by design, has numerous layers of configuration options and modes of operation.  When the verification effort gets underway and the coverage holes start appearing, even the most creative and thorough UVM testbench architect can be bogged down devising new tests – either constrained-random or even highly directed tests — to reach uncovered areas.

At the recent ARM® Techcon, Nguyen Le, a Principal Design Verification Engineer in the Interactive Entertainment Business Unit at Microsoft Corp. documented a real world case study on this exact situation.  Specifically, in the paper titled “Advanced Verification Management and Coverage Closure Techniques”, Nguyen outlined his initial pain in verification management and improving cover closure metrics, and how he conquered both these challenges – speeding up his regression run time by 3x, while simultaneously moving the overall coverage needle up to 97%, and saving 4 man-months in the process!  Here are the highlights:

* DUT in question
— SoC with multi-million gate internal IP blocks
— Consumer electronics end-market = very high volume production = very high cost of failure!

* Verification flow
— Constrained-random, coverage driven approach using UVM, with IP block-level testbenches as well as  SoC level
— Rigorous testplan requirements tracking, supported by a variety of coverage metrics including functional coverage with SystemVerilog covergroups, assertion coverage with SVA covers, and code coverage on statements, Branches, Expressions, Conditions, and FSMs

* Sign-off requirements
— All test requirements tracked through to completion
— 100% functional, assertion and code coverage

* Pain points
— Code coverage: code coverage holes can come from a variety of expected and unforeseen sources: dead code can be due to unused functions in reused IP blocks, from specific configuration settings, or a bug in the code.  Given the rapid pace of the customer’s development cycle, it’s all too easy for dead code to slip into the DUT due to the frequent changes in the RTL, or due to different interpretations of the spec.  “Unexplainably dead” code coverage areas were manually inspected, and the exclusions for properly unreachable code were manually addressed with the addition of pragmas.  Both procedures were time consuming and error prone
— Verification management: the verification cycle and the generated data were managed through manually-maintained scripting.  Optimizing the results display, throughput, and tool control became a growing maintenance burden.

* New automation
— Questa Verification Manager: built around the Unified Coverage Database (UCDB) standard, the tool supports a dynamic verification plan cross-linked with the functional coverage points and code coverage of the DUT.  In this way the dispersed project teams now had a unified view which told them at a glance which tests were contributing the most value, and which areas of the DUT needed more attention.  In parallel, the included administrative features enabled efficient control of large regressions, merging of results, and quick triage of failures.

— Questa CoverCheck: this tool reads code coverage results from simulation in UCDB, and then leverages formal technology under-the-hood to mathematically prove that no stimulus could ever activate the code in question. If it’s OK for a given block of code to be dead due to a particular configuration choice, etc., the user can automatically generate wavers to refine the code coverage results.  Additionally, the tool can also identify segments of code that, though difficult to reach, might someday be exercised in silicon. In such cases, CoverCheck helps point the way to testbench enhancements to better reach these parts of the design.

— The above tools used in concert (along with Questasim) enabled a very straightforward coverage score improvement process as follows:
1 – Run full regression and merge the UCDB files
2 – Run Questa CoverCheck with the master UCDB created in (1)
3 – Use CoverCheck to generate exclusions for “legitimate” unreachable holes, and apply said exclusions to the UCDB
4 – Use CoverCheck to generate waveforms for reachable holes, and share these with the testbench developer(s) to refine the stimulus
5 – Report the new & improved coverage results in Verification Manager

* Results
— Automation with Verification Manager enabled Microsoft to reduce the variation of test sequences from 10x runtime down to a focused 2x variation.  Additionally, using the coverage reporting to rank and optimize their tests, they increased their regression throughput by 3x!
— With CoverCheck, the Microsoft engineers improved code coverage by 10 – 15% in most hand-coded RTL blocks, saw up to 20% coverage improvement for auto-generated RTL code, and in a matter of hours were able to increase their overall coverage number from 87% to 97%!
— Bottom-line: the customer estimated that they saved 4 man-months on one project with this process

2014 MSFT presentation at ARM Techcon -- cover check ROI

Taking a step back, success stories like this one, where automated, formal-based applications leverage the exhaustive nature of formal analysis to tame once intractable problems (which require no prior knowledge of formal or assertion-based verification), are becoming more common by the day.  In this case, Mentor’s formal-based CoverCheck is clearly the right tool for this specific verification need, literally filling in the gaps in a traditional UVM testbench verification flow.  Hence, I believe the overall moral of the story is a simple rule of thumb: when you are grappling with a “last mile problem” of unearthing all the unexpected, yet potentially damaging corner cases, consider a formal-based application as the best tool for job.  Wouldn’t you agree?

Joe Hupcey III

Reference links:

Direct link to the presentation slides:

ARM Techcon 2014 Proceedings:

Official paper citation:
Advanced Verification Management and Coverage Closure Techniques, Nguyen Le, Microsoft; Harsh Patel, Roger Sabbagh, Darron May, Josef Derner, Mentor Graphics

, , , , , , , , , , ,

31 October, 2012

Ready for 100 billion “things” connected by the Internet?

The IEEE Standards Association (SA) Corporate Advisory Group (CAG) has been working to bring industry input into the standards development organization on the emerging Internet of Things (IoT) trend that will connect billions of devices with each other.

As you can imagine, the impact this will have to the service structure down to the development of connected devices will have impact on tools used to create, verify and test them from the EDA industry to the protocols that will need to be in place to facilitate this.

This past summer the oneM2M was launched to bring some groups together who were dedicated to product technical specification for the M2M Service Layer.  The impact on the IEEE, that is responsible for ongoing Internet standardization, is likewise large and not totally known.

Segars ARM Techcon Presentation smlI was reminded of the IoT impact this week by ARM’s EVP, Simon Segars.  His ARM Techcon keynote presentation this week.  noted the IoT is a merging of our digital and physical worlds.  He also said predictions are the data from smartphones is “exploding at a 100% growth rate a year for the next 4-5 years.”   To make the point even more stunning, Simon shared that Facebook expects 1-2 billion pictures will be taken and uploaded to their website around Halloween 2012.  The good news for those who did not have the time to make it to Santa Clara, CA USA for ARM Techcon, his presentation has been made available for viewing on YouTube.  You can find it here.

The IoT conversation continues around the globe.

IEEE IoT Workshop: You are invited!

ieee_mb_blue IEEE has restored service to their Internet connection at  However, connection from IEEE staff locations is tentative due to the widespread devastation of Hurricane Sandy in the New Jersey USA area where they live and work.  There may be delays in getting official invitations out on the IoT workshop.  The IEEE workshop on Internet of Things has been put together in conjunction with several of the CAG member companies, with direct leadership from our STMicroelectonics representative and input from representatives from Broadcom, GE Medical, Ericsson, Qualcomm and others. The IEEE SA staff and IoT Workshop leadership have asked those who are connected to share workshop information.  I am doing that here.

You are invited to attend and participate in the workshop.  Details on the event are:

Event Description:

The event will feature a combination of keynote speeches, product showcase and panel sessions with the goal to:

  • identify collaboration opportunities and standardization gaps related to IoT
  • help industry foster the growth of IoT markets;
  • leverage IEEE’s value and platform for IoT industry-wide consensus development,; and
  • help industry with the creation of a vibrant IoT ecosystem.

Date: 13 November 2012
Location: Milan, Italy
Fee: Free

Keynotes include:

  • Service Provider’s View of the IoT World (SP)
  • End to End Systems Security (ST)
  • IEEE-SA – Perfect Platform for the New Millennia of Consensus Development

Panel Topics include:

  • GW as an Enabler of the New Services in the IoT World
  • Monetizing Services in the IoT World
  • Security in the IoT World
  • Standard, what we have and what is missing, convergence in the technology world, collaboration opportunities.


31 October 2012 4:25 p.m. PDT
Access to has been restored.  That was quick! You can now access IoT Workshop details from IEEE directly.
31 October 2012 3:00 p.m. PDT
Due to the impact of Hurricane Sandy, power to IEEE servers has been lost and backup power sources have been depleted.  Access to the IEEE website for more information, registration and additional details is not available at this moment.  The workshop will be held.If the servers return to the Internet, I will update this notice.And if their absence appears to be something that will last longer than another day or so, I will update this blog with alternate contact information for those who would like more detailed information on how to register and where to go to attend the event. 

, , , , , ,

@dennisbrophy tweets

Follow dennisbrophy

@dave_59 tweets

Follow dave_59

@jhupcey tweets

  • #ARM now hiring formal verification engineers in Austin: exciting tech challenge + Ram is a great guy to work with.…
  • Attention all SF Bay Area formal practitioners: next week Wednesday 7/26 on Mentor's Fremont campus the Verificatio…
  • This is a very hands-on, creative role for a verification expert -- join us!

Follow jhupcey