Format Listening Tests:
Guy Churchill ([email protected])
CD, MD (ATRAC 4.5 & 3.0), MP3, VQF,
Fri, 19 Mar 1999 16:12:22 +0800
Initial impressions and final report on A/B listening tests
NOTICE and DISCLAIMER. Everything expressed here is *my* opinion.
Processes were dealt with as scientifically as possible (though some
may want to dispute that), but listening is purely subjective. You
may happen to find different results than me, I accept that, but I
do not believe you can truly tell the difference between each format
unless you conduct an experiment like this one. The disc is available
to anyone else who wants to make up their own mind on the subject.
But bear in mind however that I am not likely to entertain criticisms
such as "you should have used co-ax, because it is better" or
"Mitsubishi CDRs are superior to Kodak CDRs", or "You should be
using a full Krell system as the listening equipment", etc etc
CD, MD (ATRAC 4.5 & 3.0), MP3 , VQF, RM.
At the beginning of 1999 I requested suggestions for songs to include
in an audio compression listening test. I finally ended up choosing
"Helene" by Roch Voisine and "The Carpenter" by Thursday Diva. Both
are very clear DDD recordings, and contain a mix of simple and complex
music, varying in frequency content. I finished preparing the
recordings a few weeks later and performed initial listening
tests. Subsequently I used 2 identical CD players to perform A/B
Six encodings of each track were considered:
- The original CD
- RM - RealMedia encoded with RM encoder
- ATRAC 4.5 (from a JE-520)
- VQF - encoded with Yamaha VQF encoder at 96kbps
- ATRAC 3.0 (from a MZ-R3)
- MP3 - encoded with XingMpeg at 128kbps
All processes remained in the digital domain (optically cabled for those interested
in the optical vs. coax debate).
Equipment used for recording:
- Pentium PC with Matsushita 7502B SCSI CDR burner
- AdB Multiwave Pro24 Digital Soundcard (does full duplex recording and
- Kodak Infoguard Blank CDRs.
Equipment used for playback
CD player and DAC used for initial test listening
- Speakers: Mission 753F Bi-wired.
- Power Amps: Cyrus Power with Cyrus PSX-R DC power supply
- Pre-amp: Onkyo 828 THX receiver
CD players used for A/B switching tests.
Accessories and cables.
- DAC: Audio Alchemy
- CD: Marantz CD63se
- Speaker cables: Taralabs Space n Time 2+2
- Interconnects: Audioquest Ruby II, Siltech HF-8 Si digital interconnect.
Listening tests, Part I
Initial impressions (based on basic listening, skipping from
track to track, not A/B switching).
As it should be -- it is the original after all!
Unable to distinguishable with accuracy from the original at
more listening needed (perhaps A/B switching will show something up).
Both extreme high and low frequencies appear to be intact. No loss in
(digital sound meter peaked at the exact same level).
Small decrease in imaging, "space" around instruments not as
sound is more 2 dimensional, i.e. lacking some depth. Both extreme high
and low frequencies still appear to be intact. Loss in dB was less
1dB (digital sound meter peaked at just below the same level, less then 1%
Impressively good but instantly recognisable, decrease in image,
rates far too fast, highs washed out. Lows still there but lack "oomph"
compared to CD
Again, impressively good but instantly recognisable, sounds better then MP3
but uses less bandwidth. Highs are brighter and lows similar to MP3 .
VQF is basically the basis for MP4.
Worst of the bunch, handles the "Carpenter" track OK but the male vocals
on Helene are really poor. Loss in dB and high and low frequencies, compressed
Listening tests, Part II
Results of A/B listening tests conducted with identical
players & interconnects.
Having 2 identical CD players meant that a single remote could be used
to control both simultaneously, allowing easy navigation to tracks of
different formats. I had an accomplice start the two CD players on
different tracks, then I simply switched audio sources to make
comparisons (the A/B switching was also done via remote and the time
delay incurred when switching sources was barely noticeable). With
this setup I could cause both machines to skip backwards to the start
of the track, or fast forward at the same rate, and then go to the
next track. Using this method I was able to conduct the tests in
essentially "blind" fashion. I propose that if anyone else wants to
listen to the test CD, that they first compare the tracks without
foreknowledge of which track is which. After they have made their
comparison I will release the specific track information to them.
The first objective was to isolate parts of the track that could theoretically
sound different in each format. I found these by carefully listening to
track as a whole and marking down points of interest (or music that could
anomalies). Using the same remote I could skip back on both CD players
simultaneously and re-listen to each section as many times as I liked. Each
could then be compared to the other and later a random selection of tracks
The CD original was certainly the best, not only was the entire frequency
range still intact, the imaging was right and the individual sounds had
clarity and position.
ATRAC 4.5 was initially only distinguishable via A/B switching and
with about 10 listenings between the two. The areas where it was
different were in decay rates, depth, & point percussion. These may
not be the terms normally used, so it is important that I
explain what I mean by each one.
- Decay rates This refers to the way sounds taper off.
String instruments especially have a particular way that the note
fades over time and ATRAC seems to cut these notes short. It is a very
small amount, but noticeable in slower, quiet instrumental pieces.
- Depth Musical imaging is not one dimensional (i.e. left to
right). Objects in the audio scene appear in front of, behind, above
and below the listener as well. Music is 3 dimensional. The
positioning of instruments is a combination of speaker positioning,
listener placement and equipment type (and of course original
recording quality). ATRAC 4.5 can cause the image to narrow i.e. the
sound stage shrinks in size, yielding a little less width from left to
right, and a little less depth from front to back.
- Point percussion Percussion instruments have a multitude of
stages during their development. Take for instance the hitting of a
drum skin; there is the initial impact that causes a sound, then the
skin resonates and the sound fades off. ATRAC 4.5 handles these
transients very well, however there were two occasions where a
difference could be determined. There is an instrument that uses a
long bamboo or wood pipe with wooden rods cutting across the
mid-section, into this, small balls (marbles, sand?) are fed -- the
sound is very unique. With ATRAC 4.5 the individual attacks on this
instrument are not as distinguishable as on the CD. Also, cymbal
crashes with ATRAC 4.5 lose some initial energy.
ATRAC 3 was easily distinguished. In comparison to 4.5 it is
quite compressed image wise -- it sounds flat and lifeless. Against
CD, ATRAC 3.0 is certainly poorer -- it is muffled and lifeless (there
is that word again). Though I challenge anyone to pick ATRAC 3.0 just
by listening to it (without A/B switching), A/B switching gave a clear
result, and anyone should be able to tell the difference.
MP3 was the disappointment I expected it to be. This does not mean it
has no application - for the PC, MP3 is ideal. It has a good
compression factor, allows easy transfer over the Internet, and sets
new standards in audio compression (I said new - not better). MP3 is
instantly recognised without an A/B switching test. It introduces
some distortion to highs in particular. Also, in complex passages the
instruments get lost - you cannot follow each one - it is as if the
encoder could not work out what parts to keep and which to throw away.
This hypothesis was confirmed when I went back and recorded the same
songs at higher MP3 bitrates (128kbps was the one I used initially, as
it is the most widely used today). At 256kbps and 320kbps there were
some remarkable improvements.
I do believe MP3 will stay mainly in the computing field, having said
that have a look at the self contained MP3 car audio storage system at
empeg.com. This is
one application that certainly has promise (it would be even better if
it accepted MP3 encoded CDs, as you get 150 to 180 MP3s(!) on a single
disc). For car applications, background music, or played over typical
speakers found on most PC systems, MP3 is fine. As a serious audio
storage medium to be used with quality equipment however, I cannot
recommend it at all.
VQF Despite the even lower bitrate, Yamaha has come up with
a great little format. Once again however, not even A/B switching was
needed to distinguish it. It suffers from all the problems of
ATRAC 3.0 and MP3. How does it stack up against MP3? Remarkably
well. I was impressed at the frequency range it held. My success rate
at picking MP3 from VQF was only just better then chance (about 65%)
which leaves me to wonder, was it just chance? I think I will need to
conduct this experiment in the future and compare MP3 to VQF again. At a size
30% smaller then MP3 I will give it the nod (but it is not a widely
accepted format and the encoder is very slow -- 2 big negatives
in my book).
RealAudio I am wondering why I even tested this, it was by
far the worst, like listening to the radio (actually, radio is
better). Its voice audio application on the Internet is where it
will remain, since that is what it was designed for (and because that
is all its audio quality is good for). Bringing it into this test was
not a fair challenge.
ATRAC 4.5 is remarkably good, and for applications like portables and
car stereos I am more then happy with its performance (I have been
using an MZ-R3 since 1995 and also listen with an MZ-E25). The A/B
listening test was the only way of picking up the differences, and
only certain passages of music tripped it up. For home recording of
tapes, radio and CDs it is perfectly fine and without high quality
partnering equipment it would be impossible to tell it from CD.
Incidentally, I did some tests to compare different CD players and
DACs, and when playing the same track I could hear differences - not
all CD players and DACs are the same (DACs appear to make the major
Even though ATRAC 4.5 has far surpassed the sound quality of ATRAC 1,
MiniDisc made an impressive debut in 1992. The convenience of the MD
format far outweighs tape, and MD's lack of acceptance in the early
years was not an issue about quality, it was a combination of Sony's
poor strategy at promoting the product, high equipment and disc
prices, DCC confusion, and a DAT fraternity defending the little
territory they had. And as MD changed the face of consumer audio
recording, MP3 has done likewise with computer audio recording and
storage. But as MP3 quality stands now, its place remains firmly in
PC and network applications (not to mention the problems brought on by
copyright issues). VQF and RealAudio are further audio compression
algorithms, with their own strenghts and weaknesses.
The next project I have planned is to compare the audio quality of
Sony's and Sharp's ATRAC.
This concludes my article. Once again I invite anyone (golden-ears
especially) to listen to these recordings and report your
findings on the real differences. If you are interested, please
contact me at [email protected]
and arrangements can be made.
Appendix: CD Selections
Artist: Thursday Diva, Album: Follow Me, Track 9: The Carpenter,
Digital Music Productions 1995, Ref: CD-509
Artist: Roch Voisine, Album: Helene, Track 1: Helene, Les Disques Star
Inc. 1989, Ref: STR-CD-8014