Posts Tagged ‘functional verification’

22 July, 2017

There’s a wonderful quote in Brian Kernighan book The Elements of Programming Style, where he says “Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?” Humor aside, our 2016 Wilson Research Group study supports the claim that debugging is a challenge today, where most projects spend more time involved in debugging than any other task.

One insidious aspect of debugging is that it is unpredictable if not properly managed. For example, from a management perspective, it is easy to gather metrics on various processes involved in the product development lifecycle from previous projects in order to plan for the future. However, the unpredictability of debugging becomes a manager’s nightmare. Guidelines for efficient debugging are required to improve productivity.

In reality, debugging is required in every process that spans a products development lifecycle, from conception, architectural design, implementation, post-silicon, and even the test and testbenches we create to verify/validate a design. The emergence of SystemVerilog and UVM has necessitated the development of new debugging skills and tools. Traditional debugging approaches that were adopted for simple RTL testbenches have become less productive. To address this dearth of new debugging process knowledge I’m excited to announce that the Verification Academy has just released a new UVM debugging course. This course consists of five video sessions. Topics that are covered in this course include:

• A historical perspective on the general debugging problem
• Ways to effectively debug memory leaks in UVM environments
• How to debug connectivity issues between UVM components
• Guidelines to effectively debug common issues related to UVM phases
• Methods to debug common issues with the UVM configuration database

As always, this course is available free to you. To learn more about our new Verification Academy debugging course, as well as all our other video courses, please visit www.verificationacademy.com.

, , , , , ,

24 February, 2017

My last blog post was written a few years ago before attending a conference when I was reminiscing about the 10-year history of SystemVerilog. Now I’m writing about going to another conference, DVCon, and being part of a panel reminiscing about the 15-year history of SystemVerilog and envisioning its future. My history with SystemVerilog goes back much further.

Soul of a New MachineMy first job out of college was working with a group of Data General (DG) engineers made public in the book, Soul of a New Machine, by Tracy Kidder in 1981. Part of my job was writing a program that simulated the microcode of the CPUs we designed. Back then, there were no hardware description languages and you had to hard code everything for each CPU. If you were lucky you could reuse some of the code for the user interface between projects. Later, DG came up with a somewhat more general-purpose simulation language. It was general-purpose in the sense that it could be used for a wider range of projects based on the way DG developed hardware. But getting it to work in another company’s environment would have been a challenge.  By the way, Badru Agarwala was the DG simulation developer I worked with who later founded the Verilog simulation companies Frontline and Axiom. He now manages the Calypto division at Mentor Graphics.

Many other processor companies like DEC, IBM and Intel had their own in-house simulation languages or were in the process of developing one because no commercially viable technologies existed. Eventually, Phil Moorby at Gateway Design began developing the Verilog language and simulator. One of the benefits of having an independent language, although not an official standard yet, was you could now share or bring in models from outside your company. This includes being able to hand off a Verilog netlist to another company for manufacturing. Another benefit was that companies could now focus on the design and verification of their products instead of the design and verification of tools that design and verify their products.

I evaluated Verilog in its first year of release back in 1985/1986. DG decided not to adopt Verilog at that point, but I liked it so much I left DG and joined Gateway Design as one of its first application engineers. Dropping another name, Karen Bartleson was one of my first customers as a CAD manager working at UTMC. She recently took the office of President and CEO of the IEEE.

IEEEFast forward to the next decade, when Verilog became standardized as IEEE 1364-1995. But by then it had already lost ground in the verification space. Companies went back to developing their own in-house verification solutions. Sun Microsystems developed Vera and later released it as a commercial product marketed by Systems Science. Arturo Salz was one of its developers and will be on the DVCon panel with me as well. Specman was developed for National Semiconductor and a few other clients and later marketed by Verisity. Once again, we had the problem of competing independent languages and therefore limiting the ability to share or acquire verification models. So, in 1999, a few Gateway alums and others formed a startup which I joined a year later hoping to consolidate design and verification back into one standard language. That language was SUPERLOG and became the starting point for the Accellera SystemVerilog 3.0 standard in 2002, fifteen years ago.

IEEE Std 1800-2012There are many dates you could pick for the start of SystemVerilog. You could claim it couldn’t exist until there was a simulator supporting some of the features in the standard. For me, it started when I first read an early Verilog Language Reference Manual and executed my first initial block 31 years ago. I’ve been using Verilog almost every day since. And now all of Verilog is part of SystemVerilog. I’ve been so much a part of the development of the language from its early beginnings; that’s why some of my colleagues call me “The Walking LRM”. Luckily, I don’t dream about it. I hope I never get called “The Sleeping LRM”.

So, what’s next for SystemVerilog? Are we going to repeat the cycle of fragmentation and re-consolidation? Various extensions have already begun showing up in different implementations. SystemVerilog has become so complex that no one can keep a good portion of it in their head anymore. It is very difficult to remove anything once it is in the LRM. Should we start over? We tried to do that with SUPERLOG, but no one adopted it until it was made fully backward compatible with Verilog.

The Universal Verification Methodology (UVM) was designed to cut down the complexity of learning and using SystemVerilog. There are now a growing number of sub-methodologies for using the UVM because the UVM itself has exploded in complexity  (UVM Framework and Easier UVM to name a couple). I have also taken my own approach when teaching people SystemVerilog by showing a minimal subset (See my SystemVerilog OOP for UVM Verification course).

DVConI do believe in the KISS principle of Engineering and I hope that is what prevails in the next version of SystemVerilog, whether we start over or just make refinements. I hope you will be able to join the discussion with others and me at the DVCon panel next week, or in the forums in the Verification Academy, or on Twitter.

-Dave

, , , , , , , , , , , ,

3 January, 2017

Deeper Dive into First Silicon Success and Safety Critical Designs

This blog is a continuation of a series of blogs related to the 2016 Wilson Research Group Functional Verification Study (click here).  In my previous blog (click here), I presented verification results in terms of schedules, number of required spins, and classification of functional bugs. In this blog, I conclude the series by having a little fun with some of the findings from our recent study.

You might recall from our 2014 study we did a deeper dive into the findings made a non-intuitive observation related to design size and first silicon success. That is, the smaller the design the less likelihood of achieving first silicon success (see 2014 conclusion blog for details). This observation still holds true in 2016.

For our 2016 study, we decided to do a deeper dive related to the following:

  1. Verification maturity and silicon success, and
  2. Safety critical designs and silicon successes.

Verification Maturity and Silicon Success

Figure 1 presents the findings for ASIC/IC first silicon success, and Figure 2 presents the findings for FPGA non-trivial bug escapes into production. It is important to note that only 33 percent of ASIC/IC projects are able to achieve first silicon success, and only 22 percent of FPGA projects go into production without a non-trivial bug.

BLOG-2016-WRG-figure-Con-1

Figure 1. ASIC/IC required spins before final production

BLOG-2016-WRG-figure-Con-2

Figure 2. FPGA non-trivial bug escapes into production

A question worth asking is if there might be some correlation between project success (in terms of preventing bugs) and verification maturity. To answer this question we looked at verification technology adoption trends related to a project’s silicon success.

Figure 3 presents the adoption of various verification techniques related to ASIC/IC projects, and then correlates these results against achieving first silicon success. The data suggest that the more mature an ASIC/IC project is in its adoption of verification technology the more likelihood of achieving first silicon success.

BLOG-2016-WRG-figure-Con-3

Figure 3. ASIC/IC spins and verification maturity.

Similarly, in Figure 4 we examine the adoption of various verification techniques on FPGA projects, and then correlate these results against preventing non-trivial bug escapes into production. Again, the data suggest that the more mature an FPGA project is in its adoption of verification technology the more likelihood that a non-trivial bug will not escape into production.

BLOG-2016-WRG-figure-Con-4

Figure 4. FPGA non-trivial bug escapes into production and verification maturity.

Safety Critical Designs and Silicon Success

The second aspect of our 2016 study that we decided to examine a little deeper relates to safety critical designs and their silicon success. Intuitively, one might think that the rigid and structured process required to adhere to one of the safety critical development processes (such as, DO-254 for mil/aero, ISO 26262 for automotive, IEC 60601 for medical, etc.) would yield higher quality in terms of preventing bugs and achieving silicon success.

Figure 5 shows the percentage of ASIC/IC and FPGA projects that claimed to be working on a safety critical design.

BLOG-2016-WRG-figure-Con-5

Figure 5. Percentage of projects working on safety critical designs

Keep in mind that the findings in Figure 5 do not represent volume in terms of silicon production—the data represents projects that claim to work under one of the safety critical development process standards.  Using this partition between projects working on non-safety critical and safety critical designs we decided to see how these two classes of projects compared in terms of preventing bugs.

Figure 6 compares the number of required spins for both safety critical and non-safety critical ASIC/IC designs. While Figure 7 compares the FPGA designs with non-trivial bug escapes for both safety critical and non-safety critical designs.

BLOG-2016-WRG-figure-Con-6

Figure 6. Requires ASIC/IC spins for safety critical vs. non-safety critical designs

 BLOG-2016-WRG-figure-Con-7

Figure 7. Non-trivial bug escapes for safety critical vs. non-safety critical FPGA designs

Clearly, the data suggest that a development process adopted to ensure safety does not necessarily ensure quality. Perhaps this is non-intuitive. However, to be fair, many of the safety critical features implemented in today’s designs are quite complex and increase the verification burden.

This concludes the findings from the 2016 Wilson Research Group Study.

Quick links to the 2016 Wilson Research Group Study results

, , , , , ,

2 December, 2016

ASIC/IC Verification Results

This blog is a continuation of a series of blogs related to the 2016 Wilson Research Group Functional Verification Study (click here).  In my previous blog (click here), I provided data related to designs that actively manage power. In this blog, I present verification results findings in terms of schedules, number of required spins, and classification of functional bugs.

BLOG-2016-WRG-figure-12-1

Figure 1. Design Completion Compared to Original Schedule

Figure 1 presents the design completion time compared to the project’s original schedule. While the data suggest that in 2014 there might have been a slight improvement in projects meeting their original schedule, the 2016 findings are consistent with the 2007 and 2012 studies. The bottom line is that meeting the originally planned schedule is still a challenge for most of the industry.

BLOG-2016-WRG-figure-12-2

Figure 2. Required Number of Spins

Other results trends worth examining relate to the number of spins required between the start of a project and final production. Figure 2 shows this industry trend from 2012 through 2016. Even though designs have increased in complexity, the data suggest that achieving first silicon success is not getting any worse in terms of the number of required spins before production. Still, only a little more than 30 percent of today’s projects are able to achieve first silicon success. In addition, it is worth noting that achieving second silicon success is getting worse.

Figure 3 shows various categories of flaws that are contributing to respins. Again, you might note that the sum is greater than 100 percent on this graph, which is because multiple flaws can trigger a respin.

BLOG-2016-WRG-figure-12-3

Figure 3. Types of Flaws Resulting in Respins

Logic and functional flaws remain the leading causes of respins. One observation from the 2016 study is that there seems to have been a spike in the number of respins due to power issues.

Figure 4 examines the root cause of logical or functional flaws (previously identified in Figure 3) by various categories. The data suggest design errors are the leading cause of functional flaws, and the situation is worsening. In addition, problems associated with changing, incorrect, and incomplete specifications are a common theme often voiced by many verification engineers and project managers.

BLOG-2016-WRG-figure-12-4

Figure 4. Root Cause of Functional Flaws

In my next blog (click here), I provide some concluding remarks and observations.

Quick links to the 2016 Wilson Research Group Study results

,

21 November, 2016

ASIC/IC Power Trends

This blog is a continuation of a series of blogs related to the 2016 Wilson Research Group Functional Verification Study (click here).  In my previous blog (click here), I presented our study findings on various verification language and library adoption trends. In this blog, I focus on power trends.

Today, we see that about 72 percent of design projects actively manage power with a wide variety of techniques, ranging from simple clock-gating, to complex hypervisor/OS-controlled power management schemes (see Figure 1). This is essentially unchanged from our 2014 study.

BLOG-2016-WRG-figure-11-1

Figure 1. ASIC/IC projects working on designs that actively manage power

Figure 2 shows the various aspects of power-management that design projects must verify (for those 72 percent of design projects that actively manage power). The data from our study suggest that many projects are moving to more complex power-management schemes that involve software control. This adds a new layer of complexity to a project’s verification challenge, since these more complex power management schedules often require emulation to fully verify.

BLOG-2016-WRG-figure-11-2

Figure 2. Aspects of power-managed design that are verified

Since the power intent cannot be directly described in an RTL model, alternative supporting notations have recently emerged to capture the power intent. In the 2016 study, we wanted to get a sense of where the industry stands in adopting these various notations and what we found was essentially no change from our 2014 study. For projects that actively manage power, Figure 3 shows the various standards used to describe power intent that have been adopted. Some projects are actively using multiple standards (such as different versions of UPF or a combination of CPF and UPF). That’s why the adoption results do not sum to 100 percent.

BLOG-2016-WRG-figure-11-3

Figure 3. Notation used to describe power intent

In an earlier blog in this series, I provided data that suggest a significant amount of effort is being applied to ASIC/IC functional verification. An important question the various studies have tried to answer is whether this increasing effort is paying off. In my next blog (click here), I present verification results findings in terms of schedules, number of required spins, and classification of functional bugs.

Quick links to the 2016 Wilson Research Group Study results

, , , , , ,

31 October, 2016

ASIC/IC Language and Library Adoption Trends

This blog is a continuation of a series of blogs related to the 2016 Wilson Research Group Functional Verification Study (click here).  In my previous blog (click here), I focused on I various verification technology adoption trends. In this blog I plan to discuss various ASIC/IC language and library adoption trends..

Figure 1 shows the adoption trends for languages used to create RTL designs. Essentially, the adoption rates for all languages used to create RTL designs is projected to be either declining or flat over the next year.

BLOG-2016-WRG-figure-10-1

Figure 1. ASIC/IC Languages Used for RTL Design

As previously noted, the reason some of the results sum to more than 100 percent is that some projects are using multiple languages; thus, individual projects can have multiple answers.

Figure 2 shows the adoption trends for languages used to create ASIC/IC testbenches. Essentially, the adoption rates for all languages used to create testbenches are either declining or flat. Furthermore, the data suggest that SystemVerilog adoption is starting to saturate or level off in the mid-70s range.

BLOG-2016-WRG-figure-10-2

Figure 2. ASIC/IC Languages Used for  Verification (Testbenches)

Figure 3 shows the adoption trends for various ASIC/IC testbench methodologies built using class libraries.

BLOG-2016-WRG-figure-10-3

Figure 3. ASIC/IC Methodologies and Testbench Base-Class Libraries

Here we see a decline in adoption of all methodologies and class libraries with the exception of Accellera’s UVM, whose adoption continued to increase between 2014 and 2016. Furthermore, our study revealed that UVM is projected to continue its growth over the next year. However, like SystemVerlog, it will likely start to level off in the mid- to upper-70 percent range.

Figure 4 shows the ASIC/IC industry adoption trends for various assertion languages, and again, SystemVerilog Assertions seems to have saturated or leveled off.

BLOG-2016-WRG-figure-10-4

Figure 4. ASIC/IC Assertion Language Adoption

In my next blog (click here) I plan to present the ASIC/IC design and verification power trends.

Quick links to the 2016 Wilson Research Group Study results

, , , , , , , , , , , , , , , , , , ,

4 October, 2016

ASIC/IC Resource Trends

This blog is a continuation of a series of blogs related to the 2016 Wilson Research Group Functional Verification Study (click here).  In my previous blog (click here), I focused on FPGA design and verification trends.  I now will shift the focus of this series of blogs from FPGA trends to ASIC/IC trends. In this blog, I plan to discuss the growing ASIC/IC design project resource trends due to rising design complexity.

Figure 1 shows the percentage of total project time spent in verification. As you would expect, the results are all over the spectrum; whereas, some projects spend less time in verification, other projects spend more. The average total project time spent in verification in 2016 was 55 percent, which did not change significantly from 2012 and 2014. The data suggest that although Moore’s Law continues in terms of growing design complexity the industry isn’t getting any worse in terms of project time spent in verification.

BLOG-2016-WRG-figure-8-1

Figure 1. Percentage of ASIC/IC Project Time Spent in Verification

Perhaps one of the biggest challenges in design and verification today is identifying solutions to increase productivity and control engineering headcount. To illustrate the need for productivity improvement, we discuss the trend in terms of increasing engineering headcount. Figure 2 shows the mean peak number of engineers working on a project. Again, this is an industry average since some projects have many engineers while other projects have few. You can see that the mean peak number of verification engineers today is greater than the mean peak number of design engineers. In other words, there are, on average, more verification engineers working on a project than design engineers. This situation has changed significantly since 2007.

BLOG-2016-WRG-figure-8-2

Figure 2. Mean Number of Peak Engineers per ASIC/IC Project

Another way to comprehend the impact of today’s project headcount trends is to calculate the compounded annual growth rate (CAGR) for both design and verification engineers. Between 2007 and 2016 the industry experienced a 3.6 percent CAGR for design engineers and a 10.4 percent CAGR for verification engineers, as shown in Figure 3. Although the data suggest that the demand for verification engineers is starting to slow down in 2016, clearly, the double-digit increase in required verification engineers has become a major project cost management concern, and is one indicator of growing verification effort.

BLOG-2016-WRG-figure-8-3

Figure 3. Peak Engineers CAGR per ASIC/IC Project

But verification engineers are not the only project stakeholders involved in the verification process. Design engineers spend a significant amount of their time in verification too, as shown in Figure 4. In 2014 and 2016, design engineers spent on average 53 percent of their time involved in design activities and 47 percent of their time in verification.

BLOG-2016-WRG-figure-8-4

Figure 4. Where ASIC/IC Design Engineers Spend Their Time

However, this is a reversal in the trends observed from the 2010 and 2012 studies, which indicated that design engineers spent more time in verification activities than design activities. The data suggest that design effort has risen significantly in the last four years when you take into account that: (a) design engineers are spending more time in design, and (b) there was a nine percent CAGR in required design engineers between 2012 and 2014 (shown in Figure 2), which is a steeper increase than the overall 3.7 CAGR for design engineers spanning 2007 through 2016. We will discuss a few factors that might be contributing to this increased design effort in upcoming blogs.

Figure 5 shows where verification engineers spend their time (on average). We do not show trends here since this aspect of project resources was not studied prior to 2012, and there were no significant changes in the results between 2012, 2014, and 2016. Our study found that verification engineers spend more of their time in debugging than any other activity. This needs to be an important research area whose future solutions will be necessary for improving productivity and predictability within a project.

BLOG-2016-WRG-figure-8-5

Figure 5. Where ASIC/IC Verification Engineers Spend Their Time

In my next blog (click here) I plan to discuss various ASIC/IC verification solutions adoption trends.

Quick links to the 2016 Wilson Research Group Study results

,

25 September, 2016

ASIC/IC Design Trends

This blog is a continuation of a series of blogs related to the 2016 Wilson Research Group Functional Verification Study (click here).  In my previous blog (click here), I focused on FPGA design and verification trends.  I now will shift the focus of this series of blogs from FPGA trends to ASIC/IC trends.

In this blog, I present trends related to various aspects of design to illustrate growing design complexity. Figure 1 shows the trends from the 2014 and 2016 studies in terms of active ASIC/IC design project by design sizes (gates of logic and datapath, excluding memories). Keep in mind that Figure 1 represents design projects, not silicon volume.

One interesting observation from this year’s study is that there appeared to be a large increase in design projects working on designs less than 100K gates. This was somewhat of a surprise. Perhaps it is due to a number of projects working on smaller sensor chips for IoT devices. At any rate, it is important to keep this in mind when observing some of the verification technology adoption trends. The reason is typically these very small projects do not apply advanced verification techniques, which can bias the overall industry verification technique adoption trends in some cases.

BLOG-2016-WRG-figure-7-1

Figure 1. ASIC/IC Design Sizes

The key takeaway from Figure 1 is that the electronic industry continues to move to larger designs. In fact, 31 percent of today’s design projects are working on designs over 80M gates, while 20 percent of today’s design projects are working on designs over 500M gates.

But increased design size is only one dimension of the growing complexity challenge. What has changed significantly in design since the original Collett studies is the dramatic movement to SoC class of designs. In 2004, Collett found that 52 percent of design projects were working on designs that includ one or more embedded processors. Our 2012 study found that the number of design projects working on designs with embedded processors had increased to 71 percent, and has been fairly consistent since that point in time as shown in Figure 2.

Another interesting trend is the increase in the number of embedded processes in a single SoC. For example, 49 percent of design projects today are working on designs that contain two or more embedded processors, while 16 percent of today’s designs include eight or more embedded processors. SoC class designs add a new layer of verification complexity to the verification process that did not exist with traditional non-SoC class designs due to hardware and software interactions, new coherency architectures, and the emergence of complex network on-a-chip interconnect.

BLOG-2016-WRG-figure-7-2

Figure 2. ASIC/IC Design Sizes

In addition to the increasing number of embedded processors contained within an SoC, it is not uncommon to find in the order of 120 integrated IP blocks within today’s more advanced SoCs. Many of these IP blocks have their own clocking requirements, which often present new verification challenges due to metastability issues involving signals that cross between multiple asynchronous clock domains.

In Figure 3, we see that 91 percent of all ASIC/IC design projects today are working on designs that have two or more asynchronous clock domains.

BLOG-2016-WRG-figure-7-3

Figure 3. Number of Asynchronous Clock Domains in ASIC/IC Designs

One of the challenges with verifying clock domain crossing issues is that there is a class of metastability bugs that cannot be demonstrated in simulation on an RTL model. To simulate these issues requires a gate-level model with timing, which is often not available until later stages in the design flow. However, static clock-domain crossing (CDC) verification tools have emerged as a solution used to automatically identify clock domain issues directly on an RTL model at earlier stages in the design flow.

In my next blog (click here) I plan to discuss the growing ASIC/IC design project resource trends due to rising design complexity.

Quick links to the 2016 Wilson Research Group Study results

, ,

21 September, 2016

FPGA Language and Library Trends

This blog is a continuation of a series of blogs related to the 2016 Wilson Research Group Functional Verification Study (click here).  In my previous blog (click here), I focused on FPGA verification techniques and technologies adoption trends, as identified by the 2016 Wilson Research Group study. In this blog, I’ll present FPGA design and verification language trends.

You might note that the percentage for some of the language that I present sums to more than one hundred percent. The reason for this is that many FPGA projects today use multiple languages.

FPGA RTL Design Language Adoption Trends

Let’s begin by examining the languages used for FPGA RTL design. Figure 1 shows the trends in terms of languages used for design, by comparing the 2012, 2014, and 2016 Wilson Research Group study, as well as the projected design language adoption trends within the next twelve months. Note that the language adoption is declining for most of the languages used for FPGA design with the exception of Verilog and SystemVerilog.

Also, it’s important to note that this study focused on languages used for RTL design. We have conducted a few informal studies related to languages used for architectural modeling—and it’s not too big of a surprise that we see increased adoption of C/C++ and SystemC in that space. However, since those studies have (thus far) been informal and not as rigorously executed as the Wilson Research Group study, I have decided to withhold that data until a more formal study can be executed related to architectural modeling and virtual prototyping.

BLOG-2016-WRG-figure-6-1

Figure 1. Trends in languages used for FPGA design

It’s not too big of a surprise that VHDL is the predominant language used for FPGA RTL design, although it is slowly declining when viewed as a worldwide trend. An important note here is that if you were to filter the results down by a particular market segment or region of the world, you would find different results. For example, if you only look at Europe, you would find that VHDL adoption as an FPGA design language is about 79 percent, while the world average is 62 percent. However, I believe that it is important to examine worldwide trends to get a sense of where the industry is moving in the future.

FPGA Verification Language Adoption Trends

Next, let’s look at the languages used to verify FPGA designs (that is, languages used to create simulation testbenches). Figure 2 shows the trends in terms of languages used to create simulation testbenches by comparing the 2012, 2014, and 2016 Wilson Research Group study, as well as the projected verification language adoption trends within the next twelve months.

BLOG-2016-WRG-figure-6-2

Figure 2. Trends in languages used in verification to create FPGA simulation testbenches

What is interesting in 2016 is that SystemVerilog overtook VHDL as the language of choice for building FPGA testbenches. But please note that the same comment related to design language adoption applies to verification language adoption. That is, if you were to filter the results down by a particular market segment or region of the world, you would find different results. For example, if you only look at Europe, you would find that VHDL adoption as an FPGA verification language is about 66 percent (greater than the worldwide average), while SystemVerilog adoption is 41 percent (less than the worldwide average).

FPGA Testbench Methodology Class Library Adoption Trends

Now let’s look at testbench methodology and class library adoption for FPGA designs. Figure 3 shows the trends in terms of methodology and class library adoption by comparing the 2012, 2014, and 2016 Wilson Research Group study, as well as the projected verification language adoption trends within the next twelve months.

BLOG-2016-WRG-figure-6-3

Figure 3. FPGA methodology and class library adoption trends

Today, we see a basically a flat or downward trend in terms of adoption of all testbench methodologies and class libraries with the exception of UVM, which has been growing at a healthy 10.7 percent compounded annual growth rate. The study participants were also asked what they plan to use within the next 12 months, and based on the responses, UVM is projected to increase an additional 12.5 percent.

By the way, to be fair, we did get a few write-in methodologies, such as OSVVM and UVVM that are based on VHDL. I did not list them in the previous figure since it would be difficult to predict an accurate adoption percentage. The reason for this is that they were not listed as a selection option on the original question, which resulted in a few write-in answers. Nonetheless, the data suggest that the industry momentum and focused has moved to SystemVerilog and UVM.

FPGA Assertion Language and Library Adoption Trends

Finally, let’s examine assertion language and library adoption for FPGA designs. The 2016 Wilson Research Group study found that 47 percent of all the FPGA projects have adopted assertion-based verification (ABV) as part of their verification strategy. The data presented in this section shows the assertion language and library adoption trends related to those participants who have adopted ABV.

Figure 4 shows the trends in terms of assertion language and library adoption by comparing the 2012, 2014, and 2016 Wilson Research Group study, and the projected adoption trends within the next 12 months. The adoption of SVA continues to increase, while other assertion languages and libraries are not trending at significant changes.

BLOG-2016-WRG-figure-6-4

Figure 4. Trends in assertion language and library adoption for FPGA designs

In my next blog (click here), I will shift the focus of this series of blogs and start to present the ASIC/IC findings from the 2016 Wilson Research Group Functional Verification Study.

Quick links to the 2016 Wilson Research Group Study results

, , , , , , , , , , , , , , ,

7 September, 2016

FPGA Verification Effectiveness Trends

This blog is a continuation of a series of blogs related to the 2016 Wilson Research Group Functional Verification Study (click here).  In my previous blog (click here), I focused on the amount of effort spent in FPGA verification. We have seen in previous blogs that an increasing amount of effort is being applied to FPGA functional verification. In this blog I focus on the effectiveness of verification in terms of FPGA project schedule and bug escapes.

FPGA Schedules

Figure 1 presents the design completion time compared to the project’s original schedule. What was a surprise in the 2014 findings is that we saw an improvement in the number of FPGA projects meeting schedule—compared to 2012 and 2016. This could have been an anomaly or sampling error associated with this question in 2014, as I discussed when describing confidence intervals in the prologue. However, we saw similar results when looking only at ASIC/IC designs in 2016 (which will be published in a future blog). At any rate, it does raise an interesting question that might be worth investigating.

BLOG-2016-WRG-figure-4-1

Figure 1. FPGA design completion time compared to the project’s original schedule

FPGA Lab Iterations

ASIC/IC projects track the number of required spins that occur prior to market production.  In fact, this can be a useful metric for determining the overall verification effectiveness of an ASIC/IC project.  Unfortunately, we lack such a metric for FPGA projects.  For the 2014 study, we decided to ask the question related to the average number of lab iterations required before the design went into production, and asked the same question again in 2016. The results are shown in Figure 2.

BLOG-2016-WRG-figure-4-2

Figure 2. Number of FPGA iterations in the lab

FPGA Non-Trivial Bug Escapes to Production

I have never been convinced that lab iterations is analogous to ASIC/IC respin as a verification effectiveness metric.  So for our 2016 study we decided to also ask the question concerning the number of non-trivial bugs that escape into production and are found in the field. The results were surprising and are presented in Figure 3. Only 22 percent of today’s FPGA design projects are able to produce designs without a non-trivial bug escaping into the final product. For many market segments (such as safety critical designs) the cost of upgrading the FPGA in the field can be huge since this requires a complete revalidation of the system.

BLOG-2016-WRG-figure-4-3

Figure 3. Number of FPGA iterations in the lab (no trend data available)

FPGA Bug classification

For the 2016 study, we asked the FPGA project participants to identify the type of flaws that were contributing to rework. In Figure 4, I show the two leading causes of rework, which are logical and functional bugs, as well as clocking bugs. The data seems to suggest that these issues are growing. Perhaps due to the design of larger and more complex FPGAs.

BLOG-2016-WRG-figure-4-4

Figure 4. Types of Flaws Resulting in FPGA Rework

In Figure 5, I show trends in terms of main contributing factors leading to logic and functional flaws—and you can see that design errors are the main cause of functional flaws.  But note that a significant amount of flaws are related to some aspect of the specification—such as changes in the specification—or incorrect or incomplete specifications. Problems associated with the specification process are a common theme I often hear when visiting FPGA customers.

BLOG-2016-WRG-figure-4-5

Figure 5. Root cause of FPGA functional flaws

In my next blog (click here), I plan on presenting the findings from our study for FPGA verification technology adoption trends.

Quick links to the 2016 Wilson Research Group Study results

, ,

@dennisbrophy tweets

Follow dennisbrophy

@dave_59 tweets

Follow dave_59

@jhupcey tweets

  • #ARM now hiring formal verification engineers in Austin: exciting tech challenge + Ram is a great guy to work with.…https://t.co/uwIXLHWqvg
  • Attention all SF Bay Area formal practitioners: next week Wednesday 7/26 on Mentor's Fremont campus the Verificatio…https://t.co/9Y0iFXJdYi
  • This is a very hands-on, creative role for a verification expert -- join us! https://t.co/jXWFGxGrpn

Follow jhupcey