This blog entry is third one of the series about “real time operating systems” in embedded applications. You can find first part here and second there.
This time I will be talking about how the “multimedia” are getting close to “real time” but they are not.
Multimedia
In the modern desktop PC, smartphones, tablets and all the “smart entertainment” crap the support for multimedia is an essential functionality.
When I was young the only multimedia on computers were some 320×200 pixel jerky animations and stereo sound generated from MFM modulator sound cards or even directly at the printer port with a help of some resistors. Playing a real TV quality movie was simply far far beyond the bandwidth of that time hard disks.
Nowadays this is not a problem.
However…
“Fast” does not mean “real time”
I do not have much experience with video streams. But I do have some with sound on PC with the DirectSound technology. It is nice. It is fast. But if You will take a closer look at it and at supporting hardware You will notice that even tough it has a high bandwidth it is not “quick”.
When playing the sound You simply prepare some data, put in into a buffer in a RAM and tell a sound library to pump it from RAM to sound card hardware. This operation is captured by an operating system audio sub-system which performs intermediate operations like re-sampling, digital volume adjusting and mixing it with sounds prepared by other applications. It then moves mixed data to an another buffer in RAM and just then pumps it to sound hardware. This “pumping” is in fact done by DMA (Direct Memory Access – a hardware used to send data between blocks of address space or I/O resources in between CPU cycles) in a sequence of burst transactions.
This lengthy path results in the fact that the information that all the buffered data were transferred from the RAM to the operating system or from RAM to the sound card does not mean that the sound was actually played. It might be, but most probably it was not.
The main difficulty I encountered, and I dare to say not only I since javax.sound is also burdened by it, was the ability to actually stop playing sound. Or even worse: to detect when it finished playing.
Fast means buffered
Most of You probably noticed that when You play a video content on Youtube or some other streaming service or by media player from disk the sound is sometimes a little off with the video. And, what is easier to observe and more frequent, that when You close the player the sound still plays for a fraction of second even tough there is no application which is playing it.
This is because the operating system is “fast” but not “quick” reacting. It can pump tones of data to the sound card without any problem. But if You would like to know when exactly the last sample of 44kHz data file was used up to move the speaker membrane…. Well…
44kHz is 22μs per sample. The human ear can without a problem catch the 500μs glitch in sound as an audible “click”. If operating system would be a “real time” one with warranted response time below 22μs then You would be able to just wait for a sound to be played and then burst the next sample.
This is exactly how would You do it with a microcontroller and interrupts. You would set a timer interrupt at 44kHz and put sample by sample to the DAC driving the speaker. Without any operating system that is. You would even do it that way in DOS era. It was so simple those days…
The modern operating systems do isolate You from such hard to understand things like interrupts so you must rely on operating system task switching, signals, events and notifications to be awoken. And they simply cannot do that always quickly enough.
Note:I do suspect that modern sound card hardware does not even have the ability to notify operating system that the sound is fully played. If it could, then why DirectSound is not having such a function? The working one, I mean, because the official specs are saying: “It is there, but it most probably will not work“. And the specs are right.
I may be very wrong here however. I simply do not know it.
What You have to do is to just buffer sound ahead, set up timer and add some head-room for the task switching delay. Usually 500ms is enough.
Surprisingly You will have more problems with stopping the sound than with starting it.
Real time means: no buffers
This is the main difference between “fast” and “real time”. Real time 44kHz sound system is capable of sampling microphone data, processing it and within 22μs passing it down directly to output speaker membrane. If microphone moves the speaker moves without more that one sample delay.
Obviously most of modern digital multimedia should not be seen as “real time” media systems. Even a simple USB digital speaker or BlueTooth set will introduce delays in at least millisecond range. Ok., I may be a bit exaggerating – You can do USB speaker with lower delay if You use USB 3.0. But You should be aware that all such digital devices needs to buffer data at the source side, push them though the cable as a pack of data, and then play them to the physical output at the correct sampling rate.
It will never ever be close to sample-by-sample interrupt driven system. And it will never ever be close to a plain, dumb old piece of copper cable.
Summary
After reading this blog entry You should understand that “damn fast” is not “real time” and that even tough multimedia do require high processing power the fact that You can play it does not necessarily means it is a “real time” system.
And one another thing. If You are going to set up a cheap PC based digital recording studio with a live play-back (I do not know a proper English word here. What do I mean is, that the recorded sound is played back life to the rest of the members of the band so they can synchronize) the You may have a real deal of troubles with digitally introduced delays.
In the next blog entry I will try to show You what is multitasking about and how it relates to RtOS.
Old coder's ramblings about software concepts and quality. 