Apples to Apples Benchmark? I Don’t Think So!

I don’t normally take the time to respond to any of the various competitive claims out there. But recently in ESNUG 483, item #2, there was a posting entitled “We recently dumped Mentor Calibre for Magma Quartz DRC/LVS” ( that I feel needs to be addressed because it is misleading. So let me lay out the facts to set the record straight. Tezzaron Semiconductor Corp., Naperville, IL, became a Calibre customer when it purchased a perpetual license in 2000. In 2001, Tezzaron declined to renew technical support, and since then has not renewed the license, so they only have access to Calibre 2001.2, which was released in early 2001.

In his submission to John Cooley’s ESNUG site, Tezzaron’s Robert Patti, stated:

“We had been using Calibre for many years, but were not satisfied with it. In particular, we were looking for:

– An ability to scale efficiently across a larger # of CPU’s and machines. Calibre is OK on a single machine, but doesn’t do well outside of that. We need to get results faster.

– Better technical support. We do highly specialized designs. DRC customization is the norm rather than an exception for us.

– Native support for TCL. This is really important, as we need to be able to customize both the runsets and tool ourselves. Other EDA tools we use are TCL based. Calibre is based on a legacy runset language, and newer design rules need a better language option.”

I’ll address each of these issues in turn.

Calibre Performance

Eight years ago Calibre’s distributed processing functionality had not yet been released. Since then, Mentor has released numerous Calibre performance innovations, including the ability to thread and distribute rule file operations (Hyperscaling), and the ability to distribute layout data (Hyper Remote). Together, these features consistently provide the best performance and scaling in the market, as demonstrated in many competitive benchmarks. Here’s the data comparing the latest version of Calibre to our 2001 performance:

Calibre Performance 2001 vs 2009

Calibre Performance 2001 vs 2009

As you can see, there is an order of magnitude performance improvement and great scalability on multiple CPUs.

Calibre Support

If you want technical support, there is none better than Mentor. Mentor is the only Five Star Support award winner in the EDA industry, and we consistently rate higher than our competitors in independently conducted annual customer surveys. As I mentioned above, Tezzaron dropped their Mentor technical support in 2001.

Native Support for TCL

Calibre has supported TCL since 2004 through the TVF syntax. Like the significant scaling improvements, this is now a standard part of the Calibre DRC/DRCH package. Calibre customers with support contracts have had TCL support in Calibre at no additional charge since 2004.

Comparison to Quartz

Mr. Patti asserts in the ESNUG submission:

“In our best ‘apples to apples’ comparison we could do with Calibre (same number of threads for both tools), we found Magma Quartz to be 2-4X faster at a lower number of threads. As the number of threads increase, the gap grows even larger. We have tested from 2 to 16 cores and find the scaling to be very good. When the final top level checks on a 24 chip ‘design’ is ~2 hours, it literally saves us days.”

Tezzaron has the 2001 version of Calibre and does not have appropriate licensing to run more than a single CPU. Since there wasn’t a Calibre version number listed in Mr Patti’s write up, nor were there any Calibre performance times, I can only assume this is a comparison of an eight year old, single-CPU version of Calibre to the current version of Quartz.

With the significant improvements in Calibre runtimes, stemming from engine optimizations, scaling improvements, and hardware and OS platform support, I’m confident that had Mr. Patti tested a current version of Calibre, the results would have been dramatically different.

For further evidence of Calibre’s performance, see the customer review in the same issue of ESUNG, “Calibre nmDRC 6X speed-up from HyperScaling and Hyper Remote” ( ).

Mr. Patti also comments:

“We’ve run our ‘design’ that’s more than 20x larger than a normal design, and Quartz with Direct Read handled it, while Calibre choked.”

Without any background information, it is unclear what this means. Quartz runs on Linux platforms. With the 2001 version of Calibre, 64bit support on similar Linux platforms was not available (64bit Linux computing products were not mainstream in 2001). As a result, Tezzaron’s runs are limited to the 2Gb limit of 32bit processing in Linux machines. Of course, the current version of Calibre supports 64-bit computing, and Calibre customers are verifying some of the largest and most complex designs in the world.

The Bottom Line

All the indications (performance, scaling, availability of TCL, license status) are that this was not a valid comparison of Magma Quartz to Calibre nmDRC. It is a comparison of Magma’s latest product to an eight-year-old version of Calibre.

Post Author

Posted December 15th, 2009, by

Post Tags

, , , , , ,

Post Comments


About John Ferguson's Blog

Will provide insight into the challenges and requirements of physical verification across multiple process nodes. We'll explore new requirements, solutions and challenges. John Ferguson's Blog


2 comments on this post | ↓ Add Your Own

Commented on February 15, 2010 at 12:20 pm
By Gene Forte

NOTE: Robert Patti, CTO and VP of Design Engineering, Tezzaron Seminconductor Corp., attempted to post this comment way back in December, but was unsuccessful. When this came to Mentor’s attention on February 10, we asked Mr. Patti to send us his comments by email so we could make sure they got posted to John’s blog. –Gene Forte, Mentor Graphics, Corporate Marketing Communications.

Hi John,

Just a couple of clarifications. First, my comments about Mentor support referred to the days when we still had support. The level of support we received wasn’t worth the steep price; that’s why we decided to drop support.

Second, although our version of Calibre is indeed the older one, several of our customers are using the current versions and they have had the run issues I described, including Calibre choking. Also, we do have the 64-bit version from 2001 running on Solaris. Further, we calibrated our results against a recently released Calibre version running on Linux at a customer site.

I’m not interested in picking a fight with Mentor, but I’m sure that you are aware (as you have access to our records) that we were interested in getting an updated version before we went to Magma. Mentor declined to let us evaluate the current version and they indicated no interest in helping support our 3D requirements. I think Mentor made it abundantly clear that, as a small customer, we didn’t merit their interest.

Magma has provided us on-going support and they fix any issues we see, generally within 24 hours. They actively support our 3D needs and those of our numerous customers. We are glad we changed; we now have an enthusiastic partner in our success.

Bob Patti

Commented on February 16, 2010 at 11:24 pm
By John Ferguson

Hello Robert,

Thank you for the response and clarification. I don’t know why you were not provided access to evaluate current Calibre code. It sounds like Mentor dropped the ball there. Calibre consistently wins all head to head competitive benchmarks, against Quartz and all other competitors, for all different styles of designs and checking. I’m confident that if we had not blown the opportunity, we would have demonstrated as such for you as well.

It’s difficult to know what may be causing the crashing you describe. In my experience a “crash” is typically the result of running out of memory. In such cases, Calibre will, of course, try to report that the cause of the failure is a memory allocation issue. Unfortunately, Calibre can only provide this information to the user if the info has been passed to Calibre from the operating system. Unfortunately, not all operating systems can consistently provide this information. As a result, it sometimes appears as a random “crash”.

There are several factors that can cause memory problems. First, Calibre’s default mode, for historical reasons, is to run in 32bit processing. That means that it will only be able to allocate 2-4Gb of Memory, depending on the hardware OS. Its important to remember to set the -64 command line switch to avoid this. Another common issue is competing processes running on the same hardware. The machine might have enough memory for the Calibre job, but if some other task is also running and taking a big chunk of the memory, there may simply not be enough to share. But, in all cases, the most important thing to consider if observing memory issues with Calibre is the rule file coding. Calibre has a rich syntax, which allows someone who creates their own rules to find several paths to give accurate results. Unfortunately, not all paths are equal when it comes to performance or memory use. Mentor support can be very helpful in identifying alternative coding techniques.

In the meantime, it seems you are successful and happy with your new solution. I hope that continues for you. If you decide in the future to take another look at Calibre and its unique functionalities and features, including Programmable ERCs, Equation-based DRC, Automated Waiver Processing, Incremental DRC, FastXOR, Advanced Device Parameter extraction, and more, please feel free to contact me directly. I’m happy to help ensure you see Calibre in its true best light.

Take care,

John Ferguson

Add Your Comment