LAMP Software Stack More Reliable than Baseline Open Source Software

March 29, 2006; 02:06 AM

Coverity, Inc., makers of the world’s most advanced and scalable source code analysis solution, today released comprehensive research results on the state of quality for many of the leading open source software projects in the world. This is the first study to use source code analysis to establish a baseline metric for software quality.

As part of the government-funded analysis, Coverity is establishing a new baseline for software quality and security in open source based on sophisticated analyses of more than 17.5 million lines of source code using the latest research from Stanford University’s Computer Science department. The LAMP stack – Linux, Apache, MySQL, and Perl/PHP/Python – showed significantly better software quality above the baseline with an average of 0.290 defects per thousand lines of code compared to an average of 0.434 for the 32 open source software projects analyzed.

The analysis is the first public result arising from a contract with the Department of Homeland Security (DHS) to improve the security and quality of software. The three-year contract, called the “Vulnerability Discovery and Remediation Open Source Hardening Project,” includes research on the latest source code analysis techniques developed by Coverity and Stanford computer scientists. The analysis identified many of the most critical types of defects found in software.

“One of the goals of our research on software quality and security is to define a baseline so that people can measure software reliability in both open source and proprietary software projects,” said Ben Chelf, CTO of Coverity. “No technology can find all bugs in software, but we have collected a critical mass of data through an automated and repeatable analysis framework to show how software quality can be concretely assessed, compared, and ultimately improved.”

The open source development model benefits from the “many eyes” approach of having many developers review source code in a process similar to a large-scale peer review. This often results in high quality code, such as the code found in the LAMP stack. One goal of Coverity’s research is to accelerate this peer review process by automatically analyzing 100 percent of the code paths for defects in each software project. To do this manually for just the Linux kernel would take over twenty-eight man years alone.

As part of the analysis, Coverity is working with open source project leaders to make Coverity’s findings useful to the open source community and to assist in applying fixes to the bugs identified.

“Coverity's static source code analysis has proven to be an effective step towards furthering the quality and security of Linux," said Andrew Morton, head maintainer of the 2.6 Linux kernel. "I welcome further contributions from Coverity to help identify defects in the Linux kernel with unprecedented speed and scalability."

"Coverity's Prevent is an invaluable tool that we've now been able to integrate into the FreeBSD Project development process with nightly source code scans,” said Robert Watson, president of the FreeBSD Foundation. “Eighty-five FreeBSD developers are now registered to review Coverity-generated bug reports, resulting in hundreds of important bug fixes, one leading to a security advisory. Coverity's contributions have significantly improved the quality of FreeBSD source code base, which is greatly appreciated by both FreeBSD developers and users."

“The peer review model used by the open source community is a very powerful one and has proven effective in creating quality software,” said David Park, a co-founder of Coverity and former Stanford University computer science researcher. “With more businesses utilizing open source software like the LAMP stack, we see a need to help decision makers understand the relative quality and security in the packages they choose to bring in house.”

Coverity will continue to perform analyses of open source projects and add new projects over time. Providing this service will ensure that every line of code in a project is given a thorough review, and the results of each scan will be made freely available to the open source project development teams to encourage quick responses.

“The results that we have discovered mark a great first step in automatically assessing the quality and security of any given code base. However, our goal is not only to measure quality and security, but to make the projects that we analyze better. By opening up our analysis results to the core developers of these open source projects, we hope to work with them to reduce the number of defects and vulnerabilities in their code bases,” said Chelf.

Coverity built a web-based system that provides updated information to the general public and to developers of open source software. The system continually downloads open source software and runs scans on the software using Coverity’s static source code analysis technology. Results are updated on a daily basis. The general public can immediately access summary results and registered project maintainers and key developers can access details on the software defects.

An updated table of summary results and access to the secure database of defects is available at http://scan.coverity.com.

An explanation of the research findings with commentary on how the baseline can be used by software developers is also available for free download at http://www.coverity.com and http://scan.coverity.com.

About Coverity
Coverity (www.coverity.com), makers of the world's most advanced and scalable source code analysis solution for pinpointing software defects and security vulnerabilities, is a privately-held company headquartered in San Francisco. Coverity was founded in 2002 by leading Stanford University computer scientists whose four-year research project resulted in a breakthrough technique to address the costliest problem in the software industry. That research breakthrough allows developers to quickly and precisely eliminate software defects and security vulnerabilities in tens of millions of lines of new or legacy code. Today, Coverity's solution is used by more than 100 leading companies to significantly improve the quality and security of their software, including Juniper Networks, Symantec/VERITAS, McAfee, Synopsys, NASA, PalmOne, Sun Microsystems and Wind River.

Coverity is a registered trademark, and Coverity Extend and Coverity Prevent are trademarks of Coverity, Inc. All other company and product names are the property of their respective owners.