От: EEMBC News
[news@eembc.org]
Отправлено: 20 марта 2004 г. 0:23
Кому:
benchpress@eembc.org
Тема: EEMBC Journal - Winter 2004
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^
EEMBC JOURNAL
^ ^ ^
^ ^ ^ ^ ^ ^ ^ ^ ^
NEWS FROM EMBEDDED MICROPROCESSOR BENCHMARK CONSORTIUM
^
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^
http://www.eembc.org
Winter 2004
^ ^ ^ ^ ^
^ ^ ^ ^ ^ ^ ^
In this issue
1. Plotting the Course -- Letter from the
President
2. From the Lab
3. New Benchmark Scores
4. News
Briefs
_________________
1. Plotting the Course -
Letter from the President
There's a 99.9% probability, since you're reading
this letter, that you're already familiar with EEMBC. As evidenced by our rate
of Web site sign-ups, it's amazing to think that this organization has grown
from having a few hundred followers to more than 15,000 registered subscribers
since 1997. We've also grown from the original 12 founding members to more than
60 members, and membership continues to grow at an unprecedented
rate.
The embedded industry has definitely realized the
value of EEMBC over the Dhrystone benchmark, but we're still working towards an
endpoint where the EEMBC benchmarks are clearly the industry standard.
In
this context, "standard" has two meanings. First, when we reach this endpoint,
and it could be soon, EEMBC's members will consider it a matter of course to
publish certified benchmark scores for all their processors and in all available
configurations and frequency levels. Today, approximately 30% of the membership
publishes scores for their processors. Scores tend to be done on flagship,
leading-edge products and not necessarily on the bread-and-butter processors.
But the number of scores continues to grow, and our online database now includes
over 250 published scores.
Second, when we become the industry standard, more
system developers will stop relying on the proprietary benchmarks of processor
vendors to prove product capabilities - although the customer's application will
always be the ultimate benchmark.
To accomplish this level of success, EEMBC must
continue to develop benchmarks with greater degrees of complexity, more
system-level interaction, and even closer resemblances to real-world code. Our
new Networking version 2.0 and Digital Entertainment benchmark suites are a
major step in this direction.
Furthermore, we must strongly encourage more system
developers to join the consortium and get involved in the benchmark development
process. In other words, we need to convince you to let us help you. We're
already seeing this expansion in our Java subcommittee with the involvement of
mobile phone and PDA manufacturers as well as the associated mobile service
providers.
EEMBC currently has many new benchmarks in
development. Our goal is to release new Automotive/Industrial, Java, Office
Automation, and Telecomm benchmarks before the year is out. Moreover, we're in
the midst of approving a method to certify power consumption measurements that
will coexist with our traditional performance measurements.
Whether you're an EEMBC member or not, I'd like to
hear from you on the direction EEMBC should take. Help us plot the course for
EEMBC so we can continue to expand our contributions to the burgeoning embedded
systems market.
Markus Levy
EEMBC President
_________________
2. From the Lab
Myth: Code
Size (Lines of Code) is a Measure of Complexity
Alan R. Weiss, EEMBC
Certification Lab and Synchromesh Computing
One of the major fallacies of software engineering
is that "lines of code matters."
In decades past, much of software
engineering hinged around the concept of "lines of code," because in grappling
with the problems of schedule, budget, and quality attainment, software
engineers believed they could derive useful metrics from measuring lines of
code.
The best way to deal with these issues is to separate them into three
areas.
* runtime memory footprint
* development of new
code (and re-use of old code)
* maintenance of a code base
EEMBC members are directly interested in the
runtime memory footprint because memory is a substantial part of total system
cost. However, memory footprint is not related to this discussion. Rather, I'm
talking about the idea that "the more lines of code developed, the more . . . ."
How would you complete that sentence? Would you say
"the more lines of code developed, the more work was done by the software
engineer?" Or is the consequence "the more complex the code is to develop"? Or
"the more execution complexity there is"? In fact, I can construct scenarios
where the job of the software engineer is to improve efficiency by reducing
lines of code in an effort to get the compiler or assembler to reduce memory
footprint, although this sometimes backfires. As for more lines of code
inevitably leading to greater execution complexity, this has been proved
inaccurate. Conceptually, a compiler that replaces function calls and
subroutines with straight inlined code is actually reducing
complexity.
The more lines of code developed, the more it is
likely the system-under-benchmark will be stressed, and the more certain I am
that the memory subsystem will be stressed. This is even more the case with
measuring the compiled and linked, or "built" sizes, of
executables.
In developing the Version 2 EEMBC benchmarks, one
of our real goals was to exercise processor features not fully stressed in their
Version 1 predecessors and to track modern microprocessors and their
architecture improvements. The Version 2 code is much more likely to take
advantage of instruction-set extensions, for example, or feedback-directed
compilation. But really, the benchmarks are tracking what customers want
measured.
Clearly, the code size for the new suites is
substantially larger than EEMBC Version 1. But specifically, the magnitude of
this difference is much greater for the Digital Entertainment benchmarks
(DenBench) than for Networking Version 2. The reason is that DenBench includes
some big applications, while Networking focuses on improving the small kernels
to make them do more work. Networking Version 2 also pruned benchmarks that were
obsolete.
Development of new code is more orthogonal to
"development effort" than you might imagine. In the forthcoming Office
Automation Version 2, for example, ECL is seeking to reduce the number of lines
of code from the original source base for a new Ghostscript benchmark by
focusing on the device drivers required by the market.
Nevertheless, the absolute number of lines remains
a crude measure of how much effort is required to maintain code. In 1991, Oman
and Hagemeister introduced a composite metric for quantifying software
maintainability, called the Maintainability Index. As you might have guessed, it
was a composite of Halstead's Software Complexity Measurement, McCabe's
Cyclometric Complexity Metric, good old "Lines of Code," and the number of
comments (fewer comments means more time spent deciphering). Maintainability is
important to reduce support costs, but even there you have to be careful. If you
improve software reuse, and improve maintainability, by moving code in .C files
(local) into a separate library .C file (or even a .H header file), you do a lot
of good. Unfortunately, if they are "system" header files, small changes require
total rebuilds of your software, and longer testing time. In fact, the impact of
these changes is greater.
Both Barry Boehm and Sunita Devnani-Chulani have
worked on Bayesian analysis of software cost models. As such, they are in the
tradition of CoCoMo models: trying to estimate software "effort" by building
metrics.
We in the EEMBC Community should be proud of the
work we've accomplished on Version 2, but not because of lines of code. It's the
complexity that counts.
Copyright ©2004 by Alan R. Weiss. All rights
reserved. Permission to duplicate granted if article is unchanged and authorship
noted.
_________________
3. New Benchmark
Scores
Analog Devices
Blackfin ADSP-BF533 -
750-MHz
Production Silicon
Philips
TM5250 - 500-MHz
Simulation
Sun Microsystems
SuperH
SH-5
Simulation
Toshiba
MeP - 285-MHz
Simulation
_________________
4. News Briefs
At its January 21 meeting in Los Gatos, Calif., the
EEMBC Board of Directors voted in favor of having EEMBC begin discussions to
include power consumption as an additional metric in score reports based
on production silicon to go along with performance and code/data sizes. Related
motions were also approved to specify how power consumption will be measured and
how to account for active or passive cooling mechanisms. The Board will resume
its discussion of power consumption at its next meeting, taking place April
13-14 in Boston. In the meantime, EEMBC has formed two working groups to address
this topic. One working group, headed by Shay Gal-On of PMC-Sierra, will address
power measurements using hardware platforms. The second working group, headed by
David Lewis of ARC International, will concentrate on measuring power for IP
processor cores using simulation tools.
EEMBC will exhibit within the Convergence
Promotions booth at the Embedded Systems Conference and electronicaUSA trade
show being held in San Francisco March 30 through April 1. Stop by booth 2710-S
in Moscone Center, South Hall and pick up your free copy of the newly updated
EEMBC Literature Library CD, which includes score reports in easy-to-view
spreadsheet files.
EEMBC's membership continues to grow at a rapid
pace. Since the last issue of EEMBC Journal, the Consortium has gained four new
board members-Atmel, Cirrus Logic, Faraday Technology, and Transmeta-and two new
Java Subcommittee members, PalmOne and Time Warner Cable.
Alan R. Weiss, who co-founded the EEMBC
Certification Laboratories (ECL) with Markus Levy in 1997, is now the sole owner
of ECL. As of March 1, Levy has divested his shares in this for-profit company.
His role as EEMBC President continues unchanged, and Levy and Weiss continue to
work together closely to ensure the best services for EEMBC's members. Visit ECL
on the Web at http://www.ebenchmarks.com.
It's official: EEMBC has organized a voice over IP
(VoIP) benchmark working group. With packet voice being the future of voice
communications, In-Stat/ MDR projects that VoIP IC revenue will rise to $141.1
million in 2007. Although the size of this market is small by comparison to
consumer-oriented applications, the competition will warrant the use of EEMBC
benchmarks to allow equipment manufacturers to pick the highest performance and
cost-effective processors. Any members or non-members interested in
participating in this group, please contact Markus Levy.
EEMBC is a registered trademark of the Embedded
Microprocessor Benchmark Consortium. All other trademarks appearing herein are
the property of their respective owners.