Microsoft DirectX and Windows Media Technologies on Windows CE
Microsoft Corporation
June 1999
Summary: Describes how Microsoft® DirectX® and Windows Media Technologies provided by Microsoft Windows® CE are impacting the consumer device market. Includes a discussion on the latest real-time and multimedia capabilities of Windows CE platforms and a comparison between the desktop and Windows CE interfaces and semantics. (23 printed pages)
Contents
- Introduction
- Windows CE for Consumer Devices
- Windows CE Real-Time Features for Consumer Devices
- DirectX and Windows Media Technologies
- DirectDraw
- DirectSound
- DirectShow
- Windows Media Technologies
- Windows CE for Dreamcast Toolkit
- Conclusion
- For More Information
Introduction
We've all heard the slogans by now, the convergence of the television and personal computer made possible by new easy-to-use devices that will bring the Internet to the masses and become the multimedia entertainment centerpiece of the living room. So far, a lot of it has been more hype than anything else, but we have seen a few real products like WebTV going after the massive market of consumers still not "riding the wave." Many companies who have been successful with consumers in the PC world are starting to enter this new market. From the other direction, large consumer electronic companies are actively engaged in developing new kinds of advanced digital audio and video systems. Where does this put you as a developer? What opportunities are there for you and your company to provide the next killer consumer appliance or application? Most importantly, what tools do you need to make the move?
Microsoft Windows CE may be the answer. It's been a solution for a wide variety of embedded devices and now Microsoft is providing DirectX and Windows Media Technologies for it to help make the push further into the consumer device space as well. In this article, we want to cover a lot of material and highlight for the developer the new real-time and multimedia capabilities of Windows CE platforms. Along the way, we'll try and point out any differences between the desktop and Windows CE interfaces and semantics.
Windows CE For Consumer Devices
If you're an OEM, you can use Windows CE to develop flexible game consoles, Digital TV, DVD, and MSAudio players, as well as other consumer electronics providing the same consistent, high-quality performance available with current hardware-based devices. An OEM can develop a system employing low-cost, targeted hardware accelerating selected DirectX components. These capabilities are then accessed through DirectX and Windows Media Technologies. The real-time system characteristics available with Windows CE are essential for a software system to duplicate current hardware device quality and reliability. The first Windows CE product to ship with support for DirectX was the Sega Dreamcast!" game console, which shipped in Japan last fall and is shipping in North America and Europe later this year.
As a developer, you've been working with DirectX, Microsoft Win32®, and Microsoft Visual C++® for years. Why not leverage that experience for these new high-volume platforms? Microsoft is providing the tools necessary to allow you to do this, greatly expanding the products you can target with your expertise and current code base. That doesn't mean you can necessarily take your desktop PC application and move it over to a set-top box and expect it to work without problems. These consumer devices are targeted at a different kind of user whose expectations and experience are much different as well. The point is that you can still leverage the fundamentals, the APIs, the tools, the vast amount of resources (like this magazine) available to assist you. Chances are, you'll probably be able to make use of most of that existing code as well! If you're one of the nearly five million Win32 and DirectX developers out there, or you hire them, your expertise now allows you to target a virtually unlimited number of devices well beyond the confines of the PC world. Why not take that top-selling PC game or tax software package and sell it to users of advanced set-top boxes or entertainment platforms?
Windows CE supports, in addition to DirectX components, a carefully selected subset of the desktop Win32 APIs, such as Graphics Device Interface (GDI), WinInet, Telephony API (TAPI), and Sockets, plus some new APIs specific to embedded devices. Not all the desktop APIs make sense in new non-PC platforms, like the support for Win16-bit interfaces and other components are just too big to be usable in consumer devices of today. But Windows CE does provide support for in-process Component Object Model (COM) objects, structured storage, and Automation. Regardless of whether you use many of the Windows CE components or decide to write your own, you can take advantage of the standard Microsoft tools. Products like the Microsoft Windows CE Toolkit for Visual C++ allow you to develop your platform or applications for a Windows CE based platform with the same development environment used for desktop applications. Imagine writing a DirectX-based game that works on both Windows 98 and some Windows CE set-top box or platform using the same tools and much of the same code. All aspects of development, editing, compiling, debugging, code browsing, and so on are all available in the same manner as the desktop when developing for Windows CE. For example, Microsoft Windows CE Toolkit 1.0 for Dreamcast includes all the tools needed to develop for this SH4 processor-based platform in Visual C++.
When an OEM creates an embedded platform SDK, they use Microsoft Windows CE Platform Builder (http://msdn.microsoft.com/isapi/gomscom.asp?Target=/windowsce/Embedded/resources/pb.asp) to do it. Platform Builder has all the complex bits needed to put together a full platform development environment all the operating system modules, libraries, run times, compilers, and debugging tools you'll need to build a customized version of Windows CE. The current Platform Builder release supports various processors from AMD, ARM, Intel, Hitachi, Motorola, IBM, NEC, Philips, and Toshiba, with new ones being added. These range from low-cost processors for embedded systems to processors that can provide exceptional multimedia and 3D graphics performance. Windows CE provides DirectX for those processors useful for multimedia systems.
The modularity of Windows CE gives you flexibility in choosing which components your platform uses. When building an operating system image for your hardware using Platform Builder, you can tell the make tool whether you want a particular DirectX or UI component (like combo boxes), communications protocol, or file system. This kind of flexibility allows you to ship only those technologies you're actually using on your platform, saving space and reducing complexity. This modular design philosophy has been incorporated in the DirectX and media technologies. Even on the desktop, DirectX is modular. If you only use Microsoft DirectDraw®, no need to include Microsoft DirectPlay® or Microsoft DirectInput® or any other of the components that aren't required. If you want to create a cable box that only displays 2D images and text on the television screen, you don't need Microsoft Direct3D®, so you don't have to include it.
For an OEM, another cool feature of Platform Builder is that it generates code for a Windows CE x86 reference platform called CEPC. By using this platform, you can develop your software even before the hardware for your platform is available. In addition, Platform Builder supports commercially available reference platforms for other CPUs. Depending on the configuration of your platform, you may also be able to start development of device drivers using a CEPC or one of the other reference platforms. When using these platforms for example, the CEPC you first build your operating system image with all the components you need for it and boot the image using the Platform Builder Visual C++ version 6.0 cross-platform integrated development environment (IDE).
If you are a developer, you either create the appropriate Windows CE image yourself using Platform Builder or get them as part of an OEM's platform SDK. Once you've booted your image on the CEPC, you can write the application software for your device using Platform Builder Visual C++ 6.0 IDE, prototype any user interfaces, and possibly develop custom device drivers from the samples provided in Platform Builder as well. Of course, there are limits to how much you can do without the actual hardware, but being able to start development before it's available can be a major plus. Microsoft has used the CEPC and other reference platforms in the past to develop DirectX applications and browsers for set-top boxes and other consumer devices.
Windows CE Real-Time Features For Consumer Devices
What is the major difference between multimedia played on the PC and multimedia played on consumer home electronic devices, such as a CD player, VCR, or TV? It is the guaranteed, no-fail performance achieved by using fixed operation hardware! The PC system must be able to do all things reasonably well and so must make compromises to remain affordable, while the cheaper consumer hardware device needs to only do a few things well. Bridging this gap and making flexible, and potentially more powerful, software systems behave well is the goal. Designing your device to have real-time system characteristics is the key! Too often, real-time systems are confused with "interactive" systems ones having only a "fast" average response time. For a home consumer device, it is inadequate for us to only manage the average response time of a given set of tasks; instead we must look at worst-case performance, paying close attention to the overall hardware and software system. To ensure the product's performance, Windows CE for Dreamcast was created as an optimized version of the Windows CE operating system designed for the Dreamcast hardware.
Let's review several important ideas about systems. A real-time system has four major characteristics:
- The system must understand the application's time constraints and takes time into consideration. A consumer device playing back digitally stored or streamed media requires a complex set of cooperating tasks that must be jointly scheduled for completion before some deadline. Communication and coordination between these tasks, as well as any external controls and devices, must be performed in a timely and consistent manner. While playing an audio stream, data buffers must be filled and emptied by worker threads in a coordinated way at exactly the right times to avoid glitches or lost data.
- The system must be deterministic. The system must meet its time constraints and deadlines in a consistent manner and with bounded worst-case performance. A Digital TV using MPEG-2 video compression must not momentarily halt or "glitch," regardless of a peak computation load.
- The system must be reliable. A consumer device must work for days on end, without needing a reboot. It must robustly handle all manners of external and internal data errors, as well as poor environmental conditions in the home. An audio player must continue to function adequately even with reduced network bandwidth.
- The system must interact in a consistent manner with itself and with other system resources , including CPU, memory, buses, clocks, streaming multimedia audio and video hardware, 3D graphics accelerators, files, and network components. Designing a deterministic consumer device requires an overall system approach to all aspects of the design, not just CPU or memory usage.
Quality multimedia rendering requires a designed level of system "determinism." Determinism is property of the entire system, reflecting careful coordination among multiple processes, threads, and available system resources. Deterministic systems strive to optimize for the "best worst case" (a characteristic of real-time systems is they will happily exclude lower-priority processes and events), as opposed to providing the "best average case" (a characteristic of interactive systems is that they want to share system resources equally among all requestors). Deterministic systems are not open as are desktop systems. Instead, they are tightly controlled because they must plan for and limit interference among tasks potentially competing for scarce system resources.
A real-time operating system (RTOS), such as Windows CE, provides many essential system building blocks that can be used by a developer to implement a real-time system. Windows CE is a preemptively scheduled, multithreaded operating system providing flexible task scheduling, thread synchronization, virtual memory management, inter-process communications, fast, non-blocking I/O and file systems, external connectivity features, and a good integrated development and debugging environments. Because consumer devices are cost sensitive, Windows CE with its componentized build can minimize the overall memory footprint, allowing the cost of system memory to be used by useful applications and not taken away by the operating system as part of its run-time overhead.
With real-time systems, interrupt latency is an important consideration. Latency refers to the amount of time that elapses after an event occurs and before the software responds. Real-time systems are designed for speed and predictability, with a time response scaled by the characteristic time of the system. Multimedia and graphics systems generally use either the audio clock (22.67 microseconds at 44.1 kHz), video vertical synch (16.66 milliseconds at 60 Hz), or a MPEG-1/2 presentation/decode time (100 nanosecond tick system hardware clock gathered into an elementary stream packet clock reference) as their characteristic time scales. Experience has shown that a multimedia system requires that its control be much faster than its characteristic time scale. A multi-media system requires system timer, interrupt service routine, and event handling (from synchronous and asynchronous interrupts) latencies to be less than 100 microseconds. Windows CE provides interrupt service routines with bounded response times of this order. Using Windows CE, a closed consumer system can be highly optimized to ensure deterministic response times to internal and external events.
DirectX And Windows Media Technologies
DirectX gives a developer fast low-level access to a system's sound and graphics hardware while continuing to operate through the high-level features of Windows CE's Win32 interfaces. It lets you interact with video and audio hardware installed on your system in a device-independent manner. You can improve an existing device or implement a new system together with its driver, without having to rewrite your application code. In addition, by adding Microsoft DirectShow® and Windows Media Technologies, DirectX provides the developer easy APIs to stream, parse, decode, and render compressed multimedia audio and video data streams.
DirectX is built on Microsoft's Component Object Model (COM) and consists of a set of fast low-level libraries that you can call to create and use objects such as surfaces, palettes, textures, lobbies, sound buffers, and musical instruments. Layered over these low-level interfaces are APIs for implementing consumer audio and video devices, as well as broadband and Internet multimedia streaming.
We will be releasing the first Windows CE Platform Builder DirectX components this summer as part of Windows CE 2.12 DXPAK. It contains the following multimedia components:
- DirectDraw® 2D graphics, including memory management, fast bit block transfers, overlays, and flips, all compatible with GDI.
- DirectSound® capturing, mixing, and playing sounds.
- DirectShow® playback of multimedia audio and video streams and files, support for multimedia codecs.
- Windows Media® Player (WMP) Microsoft ActiveX® controls for playback of all major multimedia types using DirectShow.
Additional DirectX DXPAK components planned for future Windows CE releases include:
- DirectMusic® composition and playback of message-based musical data.
- Direct3D® Immediate Mode (IM) interactive 3D graphics and rendering.
- DirectPlay® networked multi-player gaming.
- DirectInput® personal input devices, including force feedback. (DirectInput is already available as part of the core Windows CE 2.12 release.)
- Windows Media Services (WMS) client filters and codecs multimedia streaming over the Internet, including support for low-bandwidth connections using MSAudio and MS MPEG-4 video.
Selected components of Windows CE DirectX 6 and media technologies will be available as part of Windows CE 2.12 on the X86, MIPS R4300, and SH4 CPUs. Additional DirectX media components, especially those needed for DVD and home electronics devices, and enhanced support for Windows CE CPUs, are planned for future DXPAK releases.
Windows CE for Dreamcast already contains DirectDraw, DirectSound, DirectMusic, Direct3D IM, DirectInput, and DirectPlay. These highly optimized, driver-based implementations are particular to the Sega Dreamcast hardware and are available as a separate SDK, but they are not part of the general Windows CE DXPAK releases. We will have much more to say about Dreamcast later in this article.
DirectDraw
Windows CE DirectDraw offers you fast access to your display hardware while retaining compatibility with the Windows GDI. Basically, it's a specialized memory manager for both system and display device memory, with hardware acceleration where available. This is important to you because before DirectX you couldn't access your display memory directly. With DirectDraw you can allocate and manipulate memory in both system and graphics memory, including transfers between the two. This allows you to do bit block transfers (blits), page flips (synchronize to video display vertical refresh), and overlays (place one image surface over another on the video display), and alpha destination blending (blend two surfaces using the source alpha image component). DirectDraw also serves as a building block for other DirectX components, such as Direct3D IM and DirectShow.
Windows CE 2.12 DXPAK furnishes the following DirectDraw 6.1 interfaces:
- IDirectDraw4 creates and manages DirectDraw objects: sets cooperative levels; controls the display device; gets caps; changes display mode; creates surfaces and palettes; accesses GDI surfaces.
- IDirectDrawSurface4 implements fast blts; associates palettes; locks and unlocks surface memory; supports overlays and page flipping; interfaces with GDI; attaches and detaches surfaces.
- IDirectDrawPalette assigns color mapping and colors to a palette.
- IDirectDrawClipper assigns rectangles to a surface to stencil the valid drawing area for the destination buffer.
- IDirectDrawColorControl provides TV-like color controls for a surface.
- IDirectDrawGammaControl controls display intensity gamma correction.
- IDirectDrawVideoPort gives a video stream direct access to DirectDraw hardware surface memory.
- IDirectDrawVideoPortContainer creates and manipulates video ports.
- IDirectDrawSurface5 supports AlphaBlt.
Windows CE DirectX supports the Windows DirectX interfaces. Take a look at the DirectDraw Mosquito sample in the DirectX 6.1 SDK to get a good idea of what you need to do on Windows CE platforms. The major differences between the Windows CE Mosquito source and the Windows version are Unicode strings, window issues, and using a YUV overlay surface and RGB primary surface. Windows CE uses Unicode for all string operations. For example, the Windows CE WinMain prototype is:
int APIENTRY WinMain(HINSTANCE, HINSTANCE, LPWSTR , int);where the data type for lpCmdLine is the Unicode LPWSTR when UNDER_CE is defined rather than the standard Windows LPSTR . The need to convert to Unicode strings also arises in window class names, resource names, and error messages. The window issues arise from the simple fact that while all Windows CE devices have the concept of a window, not all are actually capable of displaying one. To that end, the developer creates a window to handle messages, but will not draw it. Error handling raises the same issue. All CE devices are capable of doing something meaningful with messages passed to the OutputDebugString API. However, they might not be capable of the MessageBox API that the desktop code prefers for error handling. Finally, the Mosquito program has been changed to use the YUV color model for the overlay surfaces because many Windows CE applications involve streaming video and are best implemented using YUV mode surfaces.
Windows CE 2.12 DXPAK supports an AlphaBlt with a SrcoverDest operation, which is the most generally useful operation for compositing a source image over a destination image. The operation can be repeated to composite a series of source images in back-to-front order into a single destination surface.
One way to think about this blending operation is as follows: When viewing a background or destination pixel value (RGBdest, ALPHAdest) stored in the DirectDraw primary surface through a translucent intermediate pixel (RGBsrc, ALPHAsrc) stored in another source DirectDraw surface, the combined source over destination blended pixel value is:
RGBover = ALPHAsrc * RGBsrc + (1 ALPHAsrc) * RGBdestand:
ALPHAover= ALPHAsrc * ALPHAsrc + (1 ALPHAsrc) * ALPHAdestAlpha blending can be performed only if the destination surface has a non-palettized pixel format such as RGB or YUV. The source surface for the blit may have either a palettized or non-palettized format. You can read more about alpha for compositing in the Siggraph '84 article by Duff and Porter.
Windows CE DirectDraw interfaces are generally fully consistent with the Windows version of the APIs, but sometimes they can differ slightly. For example, surfaces are never lost in Windows CE under normal circumstances and interactions with GDI surfaces through non-GDI primaries can be different. Invalidation of Windows CE surfaces may occur when the display mode is changed or a surface in exclusive mode is created, loses or gains focus, or receives WM_SYSCOMMAND or SC_MONITORPOWER. All surfaces are invalidated on StartExclusiveMode.
Because a special interface IDirectDrawSurface5 has been added to support AlphaBlt, you will need to first create an IDirectDrawSurface4 and then QueryInterface for IDirectDrawSurface5.
// create surface from surface descriptor ddsd and ddraw object lpDD
ddrval = lpDD->CreateSurface(&ddsd, &lpDDSurface4, NULL);
If(ddrval == DD_OK)
{
// valid surface, so query for IDirectDrawSurface5
lpDDSurface4->QueryInterface(IID_IDirectDrawSurface5,
(LPVOID *) &lpDDSurface5);
}
else
{
// Surface couldn't be created
}DirectDraw 6.1 uniqueness values and multi-monitor capabilities are not supported because of the need to keep the system memory footprint as small as possible.
DirectSound
DirectSound allows you to play back audio data in a variety of formats, at various rates, and in either mono or stereo, as well as capture it with positional sound effects. It stores individual sounds in separate buffers and can mix an arbitrary number of them together simultaneously to produce the desired output. The storage and mixing are done using pulse code modulation (PCM) to optimize throughput. With DirectSound you get low-level yet device-independent access to your sound device, enhanced automatically by hardware acceleration when it's available.
Windows CE 2.12 DXPAK furnishes the following DirectSound 6.1 interfaces:
- IDirectSound creates and manages DirectSound objects: sets cooperative levels; gets caps; creates sound buffers, set speaker configurations.
- IDirectSoundBuffer creates and manipulates sound playback buffer.
- IDirectSoundCapture creates sound capture buffer.
- IDirectSoundCaptureBuffer manipulates a sound capture buffer.
- IDirectSoundNotify sets up notification events for a capture or playback buffer.
Windows CE DirectSound interfaces are generally fully consistent with the Windows version of the APIs. Some minor differences can occur because the Windows and Windows CE driver models are not identical. For example, DirectSound buffers are never lost under normal conditions. DirectSound 6.1 sound hardware memory compaction, 3D sound capabilities, and IKsProperty sets are not supported.
DirectShow
Windows CE DirectShow gives you the ability to play back multimedia streams from either local files or (when combined with Windows Media Technologies) streamed over a network. Many important multimedia types are supported: Motion Pictures Expert Group (MPEG) audio and video standards; audio-video interleaved (AVI); wave (WAV); musical instrument digital interface (MIDI); and Advanced Streaming Format (ASF).
The fundamental object of DirectShow is the filter, a modular component that can be classified generally as a source, transform, or renderer. Filters operate on data streams by reading, copying, modifying, or writing the data to a file or rendering it to an output device. They are arranged in a configuration called a filter graph, controlled by the Filter Graph Manager (FGM). A DirectShow graph (see Figure 1) consists of a directed sequence of filters connected by input and output filter pins. These filter pins negotiate which media types they will support. The FGM controls the multimedia data flow between the graph filters. Because DirectShow has a reconfigurable filter graph architecture, DirectShow can support playback of many media types using the same software components. A developer can extend DirectShow's multimedia support by writing their own filters.
Using DirectShow, it's easy to play multimedia files. Here is a fragment showing how to write a trivial multimedia file player application (among other simplifications, we've supressed QueryInterface return status).
HRESULT PlayMovie(LPTSTR lpszMovie)
{
// we will use several DirectShow interfaces
IMediaControl *pMC = NULL;
IGraphBuilder *pGB = NULL;
IMediaEventEx *pME = NULL;
long evCode; // something to hold a returned event code
// instantiate a filter graph as in-proc server
hr = CoCreateInstance(CLSID_FilterGraph, NULL, CLSCTX_INPROC,
IID_IMediaControl, (void **)&pMC);
// we'll use this interface to build the graph
hr = pMC->QueryInterface(IID_IGraphBuilder, (void **)&pGB);
// we'll want to wait for completion of the rendering, so we need a
// media event interface
hr = pMC->QueryInterface(IID_IMediaEventEx, (void **)&pME);
// now we're ready to build the filter graph based on the source file
// data types
hr = pGB->RenderFile(lpszMovie, NULL);
// play the source file
hr = pMC->Run();
// block application until video rendering operations finish
hr = pME->WaitForCompletion(INFINITE, &evCode);
// release interfaces
}CoCreateInstance is instantiating a filter graph object, but no filters (it doesn't yet know what media types it needs to playback), and returning an IMediaControl interface for running, pausing, and stopping the streaming of media through its filters (It also provides simple methods for building graphs, but we'll use IGraphBuilder to do that.) Because Windows CE currently supports only in-process COM servers, CLSCTX_INPROC_SERVER is the only valid server context for CoCreateInstance . Trying anything else will return E_NOTIMPL. IGraphBuilder is used to create a filter graph, add filters to or remove filters from a filter graph, enumerate all the filters in a filter graph, and force connections when adding a filter. We are using its RenderFile method to build the graph, where the final graph constructed depends on the video and audio formats contained in the source file. Finally, we can play back the file using IMediaController::Run . Because we want the application to wait until the rendering is finished, we've added IMediaEvent::WaitforCompletion .
Multimedia file formats supported in Windows CE 2.12 DXPAK DirectShow include:
- Audio formats WAVE, AIFF, AU, SND, MIDI, MPEG audio, and ASF
- Video formats AVI, , MPEG-1, MPEG-2, MPEG-4, and ASF
- Streaming formats ASF and ASX
- MIDI
Multimedia codecs available with Windows CE 2.12 DXPAK DirectShow include (see Table 1):
- Audio MSAudio 4.0, , Sipro Labs Telecom ACELP.net, Voxware MetaSound
- Video MS MPEG-4 V3, MPEG-1, CIT Cinepak
MSAudio, ACELP.net, and MS MPEG-4 are state-of-the-art codecs providing the best possible audio, speech, and video coding at a given bit rate. Microsoft MSAudio 4.0 is an advanced perceptual audio codec used for high-quality music at medium bit rates. It delivers near-CD quality stereo at 58-128 Kbps. Sipro Labs Telecom ACELP.net is a sophisticated algebraic code excited linear predictor (ACELP) speech codec designed for 8 16 kHz mono at 5-16 Kbps. Microsoft MPEG-4 V3 is a high-quality MPEG-4 video codec providing CIF (352x288) resolution at 28.8-300 Kbps. The other codecs support existing .asf file content on the Internet.
Other broadband and computation-intensive formats and codecs, such as MPEG-2 HDTV video, advanced audio coding (AAC), and Dolby AC-3 multi-channel audio are supported with DirectShow filters implemented using specific platform hardware features.
Table 1. Windows CE Audio and Video File Types and Codecs
Audio file typesCodecs AIFF, Au, Wave PCM, ADPCM, mu-law and A-law
MIDI N/A (uses MIDI)
MPEG-1 MS MPEG-1 audio layers
ASF, ASX MSAudio 4.0, ACELP.net, Voxware MetaSound Video file typesCodecs
AVI Cinepak
MOV Cinepak
MPEG-1 MPEG-1
MPEG-4 MPEG-4 V3
ASF, ASX MPEG-1, MS MPEG-4 V3, CinepakWindows CE DirectShow DXPAK includes the FGM that has an aggregated plug-in distributor (PID) interface to manage correct distribution and collection of stream control directives and common stream properties to and from all filters in graph. Common PID tasks include running, pausing, stopping, seeking, setting rate, and getting duration.
Windows CE DirectShow 2.12 DXPAK provides the following DX 6.1 filters:
- Source Filters File Source (Asynch and URL)
- Parsers and splitters MPEG-1 Stream Splitter, AVI Splitter, QuickTimer Parser, WAVE Parser, MIDI Parser, and ASF Parser
- Decoders MSAudio Decoder, MPEG-1 Audio Decoder, MPEG-1 Video Decoder, MPEG-4 Video Decoder, AVI Decompressor, and ACM Audio Decompressor
- Renderers Audio Renderer (using WaveOut), DirectSound Audio Renderer, MIDI Renderer (using MIDIOut), and Video Renderer (using either DirectDraw or GDI)
- Miscellaneous filters Overlay Mixer
Table 2 contains a short description of these filters.
Table 2. Windows CE DirectShow Filters
FilterDescription
ACM Audio Compressor Acts as a container for the Audio Compression Manager (ACM), integrating the ACM with the DirectShow architecture.
Audio Renderer Connects to the output of any of the following filters, if they contain WAV audio: File Source (Async), File Source (URL), MPEG-1 Stream Splitter, AVI Splitter, WAVE Parser, or any audio transform filter.
AVI Decompressor Decompresses AVI video input and generates suitable output for a video rendering filter or an intervening video transform filter.
AVI Splitter Parses AVI-compressed video data and splits it into the component data streams.
Color Space Converter Converts from one RGB color type to another, such as between 24-bit and 8-bit RGB color.
DSound Audio Renderer A generic audio rendering filter that you can connect to the output of any of the following filters, if they contain WAV audio: AVI Splitter, WAVE Parser, or any audio transform filter.
File Source (Async) A generic asynchronous source filter that works with any source file whose media major type is stream. This includes .avi, .mov, .mpeg, and .wav files.
File Source (URL) A generic asynchronous source filter that works with any source file that can be identified by a Uniform Resource Locator (URL) and whose media major type is stream. This includes .avi, .mov, .mpeg, and .wav files.
MIDI Parser Parses MIDI data from the File Source (Async) and File Source (URL) filters.
MIDI Renderer Renders MIDI data from the MIDI Parser filter.
MPEG Audio Decoder Decompresses MPEG-1 audio data.
MPEG Video Decoder Decompresses MPEG-1 video data.
MPEG-1 Stream splitter Splits MPEG-1 data into separate audio and video streams.
Quicktime Movie Parser Splits Apple QuickTime data into audio and video streams.
Video Renderer Connects to any video transform filter that produces decompressed video data. This filter has its own plug-in distributor in the Filter Graph Manager, which enables applications to set and retrieve properties on the filter by calling the corresponding interface methods on the Filter Graph Manager.
WAVE Parser Parses WAV-format audio data.Many DirectShow filter samples are provided and you can write your own DirectShow filters as needed. You can add your own codecs to DirectShow, either by writing a new decompression filter or by using the Audio Compression Manager (ACM) for audio codecs and Video Compression Manager (VCM) for video codecs. This allows you to easily add support for new media streams and codecs.
Hardware-based DirectShow filters, such as AC-3 Audio Decoder, DVD, and MPEG-2 Link-Layer, Transport Descriptor, TransportDemux, Line 21 Decoder, Stream Splitter, and Video Decoder can be implemented by an OEM for a particular platform configuration and subsequently used as any other DirectShow filters. By implementing DirectShow filters, an OEM can enable and control multimedia features such as MPEG-2 video and audio stream parsing, splitting, decode, and playback using the same sets of common interfaces and controls.
Most Windows DirectShow sample code can be easily recompiled and used in Windows CE. See the Windows DirectShow Web site at http://msdn.microsoft.com/isapi/gomscom.asp?Target=/DirectX/dxm/help/ds/default.htm for detailed information on the DirectShow architecture, interfaces, and sample filter code.
Windows Media Technologies
Windows Media Technologies (WMT) 4.0 includes Windows CE client software components and Microsoft Windows NT® Server applications. On the server side, Windows Media Services (WMS) delivers a superior multimedia experience for the user over low-bandwidth 28.8 Kbps modem connections and supports MPEG-4 streaming up to 5 Mbps on broadband networks. Windows CE WMS client software makes it easy for a developer to write applications that can access content from either a local disk or over the Internet. Using its Advanced Streaming Format (ASF), WMS can supply whatever content is appropriate for your connections. The ASF content streams for WMS can be composed of different media types, which can come from different servers. Supported multimedia file types include WAV, AVI, MPEG, and QuickTime. WMS also provides seamless switching, which means you can play any number of clips without pausing in between them.
WMS provides a developer special client DirectShow filters that support ASF streams using UDP, TCP, and HTTP protocols. It provides event-driven stream switching and smart streaming that takes into account time-dependent and site-dependent connection bandwidth variation. WMS has multi-datarate encoding, where multiple streams with different bit rates are created in one .asf file and the client negotiates with the server for the appropriate stream. It automatically adjusts the stream depending on playback conditions and can select from multiple video streams based on their bandwidth.
The client can dynamically thin the stream based on the available bandwidth using an algorithm that changes delivery from full frames to key-frame only. If necessary, the WMS client can ask the server to send only audio and no video. Audio is given top priority because it is considered critical to the user experience. As network bandwidth and conditions improve, WMS steps the video back up to restore the viewing to its optimal level. In addition, a UDP resend capability allows the client, if there is time available, to request missing packets from the server.
The Windows CE DirectShow codecs used with the WMS client software support a wide range of low- and medium-bandwidth multimedia, including streams of:
- Speech-quality 8 16 kHz mono at 5-16 Kbps (using Sipro ACELP.net).
- High-fidelity 8-48 kHz stereo at 56-128 Kbps with near FM-radio quality with 28.8 Kbps modems and near-CD quality at 64 Kbps (using MSAudio 4.0).
- Up to 30 fps QCIF (176x144)-CIF (352x288) resolution video at 28.8-300 Kbps (using MS MPEG-4).
In addition, when used with broadband networks, MS MPEG-4 supports true broadcast quality video.
A key part of WMT 4.0 and DirectShow is the Windows Media Player (WMP) control. This is an ActiveX control used by a developer to add multimedia playback capabilities to applications, as well as to Web pages. It allows you to embed Windows Media content in applications that support ActiveX controls, like Microsoft Internet Explorer, Microsoft Visual Basic® and Microsoft Visual C++® development environments. It provides a programming interface for rendering a variety of network streaming (as part of WMS) and non-streaming multimedia formats. The WMP controls use an appropriate DirectShow filter graph to parse, decode, and render the media stream. The WMP controls are implemented through the DirectShow FGM.
Windows CE WMP controls incorporate many of the features of the older Microsoft ActiveMovie!" control and the Microsoft NetShow!" Player version 2.0 control, as well as many new features. Functionality exposed by the methods and properties includes:
- Building custom user interfaces on top of the control. Buttons, boxes, and menus can be incorporated into your Web page or application for administering the control.
- VCR-like control functions, such as start, stop, pause, fast-forward, rewind, mute, and volume adjustment.
- Monitoring transport stream and station performance.
- Retrieving information about the content, such as the title, author, and copyright date.
- Universal player that plays ASF content.
- Play lists that can process a sequence of distinct media pieces into a seamless steam.
- Enhanced .asx files for branding and control of ASF duration.
Your application or Web page can also be designed to respond to event messages fired by the control. Events that trigger the control to fire an event include:
- Warnings and errors encountered by the control.
- Changes in the state of the stream, such as from a play state to a paused state.
- Keyboard and mouse input.
- The control disconnecting from a stream server.
- Script commands being received by the control.
Windows CE WMP is also provided as a stand-alone player application based on the same technology as the player control. However, the control allows you to create a user interface unique to your content, process any type of script commands, and integrate multimedia presentations into larger Web applications.
WMP plays the media types and codecs just listed for DirectShow. You can select different video image, including full screen, or resize the video to any size you want, depending on your preference and image quality.
There are two ways of using WMP to play content:
- As a stand-alone player. When you use WMP as a stand-alone application, you can decide how you want to have access to the WMT content. If you play the WMT content file locally, you are not streaming the file. WMP reads the content file from the hard drive and then plays it. If you play a content file from a remote location, it streams to your computer. You can start the stand-alone player from a link in a Web page.
- Embedded in a Web page, Microsoft JScript® application, or another ActiveX container application. An easy way of providing WMP to people who do not already have it on their computers is to embed the player in a Web page. The script commands used to embed the player can identify the WMT content to play, as well as show how to play it.
If you are designing Web pages to run on Internet Explorer using DirectX, you have possibilities that go beyond standard scripting. Smooth, tear-free animation, accompanied by high-quality mixed audio and immediate response to user actions, can make your Web pages stand out. DirectX Transform provides additional effects, such as explode, blur, fade, and ripple.
As previously mentioned, WMP is scriptable with HTML and JScript, so it can be embedded in a Web page and readily updated or customized. Because it supports all major formats, it gives you a wide range of possibilities for enhancing your Web page. One of the most powerful of these is presenting a multimedia stream while providing the user with associated links to view simultaneously. Web controls, or pages on Web sites that control playback of local media, are also easy to implement using this technology.
Windows CE 2.12 DXPAK WMP controls include a carefully selected subset of the WMP features and most ActiveMovie controls. Excluded WMP control properties are markers, optional flags, and ability to turn on/off options.
Windows CE For Dreamcast Toolkit
Microsoft and some of its partners are already using DirectX on Windows CE for several consumer devices. As we already mentioned, the first product to ship with support for Windows CE and DirectX was the Sega Dreamcast game console. Microsoft recently released the Windows CE for Dreamcast Toolkit to licensed developers. Windows CE for Dreamcast is an optimized version of the Windows CE operating system designed for the Dreamcast hardware. It includes all the tools needed to develop for this SH4 processor-based platform in Visual C++ and a highly optimized implementation of the DirectX APIs. It is a prime example of a platform where you can leverage your existing Win32 and DirectX expertise and code to develop content for new consumer devices.
In the past, console developers were forced to learn and use proprietary APIs and tools for each new platform. Developing for a new console almost always meant dealing with all the hassles of first-generation development tools. By providing Win32 and DirectX on Dreamcast, Sega and Microsoft are for the first time allowing game developers to create titles for both the PC and console using virtually the same code base, as well as providing the opportunity to leverage existing code for the Dreamcast exclusive titles. Also, you get to do all the development using the same familiar tools used for desktop games.
The Dreamcast console (shown in Figure 2) includes a 200-MHz Hitachi SH4 microprocessor and VideoLogic PowerVR graphics chip capable of rendering more than 1 million polygons per second. The system has 16 MB of main memory, 8 MB of video memory, and 2 MB of sound memory. The sound chip is the Yamaha AICA, which uses the ARM7DI as its sound controller. Unlike other consoles that either use cartridges (like the Nintendo 64) or standard CD-ROMs (like the Sony Playstation), Sega designed a proprietary storage format for the Dreamcast called GD-ROM (Gigabyte Disk ROM). These discs can store both standard CD-ROM data and game data in a high-density band. The GD-ROM format allows Dreamcast titles to store more data than standard CD-ROMs without having to go to the much more expensive DVD drives. There are currently several peripherals available for the Dreamcast, including a standard controller, microphone, light gun, wheel, and arcade-style joystick. Many of the peripherals support the use of plug-in Visual Memory System devices. These small devices serve as both persistent memory storage units as well as portable game players using their built-in LCD screen. One of the most promising features of the Dreamcast is its built-in modem. It allows you to provide the same kind of online game play features available in many PC titles today, as well as provide Internet connectivity previously unavailable on consoles.
The Dreamcast console does not actually ship with Windows CE built in. The operating system, DirectX, and the game itself are built as one image and stored on the GD-ROM. When the GD-ROM is placed in the Dreamcast console, the boot ROM loads the bootstrap code, which then loads the Windows CE operating system. This way, there are no versioning issues. You ship the version of Windows CE your game was developed on with your game, so there is no need to worry if the user has the correct version installed or not. There are, of course, differences between the Win32 and DirectX on Dreamcast compared to the desktop. There is no Windows UI (no Windows Explorer, Start menu, and so on), so the game is in charge of providing the entire user interface. There is no support for multiple windows either; the game runs in full-screen exclusive mode with DirectDraw and Direct3D as the primary graphics APIs. Like other Windows CE platforms, you can pick and choose which components you wish to build with your game. For example, if your game does not make any use of the networking features of Dreamcast, you don't have to include DirectPlay or Winsock.
Because Windows CE does not provide its own UI for the Dreamcast, much of the functionality of the Win32 user component as been removed. Only the APIs for accessing resources and some user input WM_* messages are supported. Specifically, those actually dealing with window elements like buttons or menus are removed. Similarly, much of the GDI has been removed because DirectDraw and Direct3D are the main graphics APIs. What's left of the GDI is support for loading fonts and bitmaps. A new Dreamcast-specific API has been introduced that allows read/write permission to the Visual Memory System devices mentioned earlier. For using the Dreamcast communication features, the standard Win32 serial communications API is supported, as well as Winsock 1.1.
Windows CE for Dreamcast supports the full set of DirectX foundation components. DirectDraw, Direct3D Immediate Mode, DirectInput, DirectSound, DirectPlay, and DirectShow are all supported. An upcoming release will also add support for DirectMusic. All these components have been optimized for Dreamcast to improve performance, reduce size, and take advantage of platform-specific features. This is done while still providing developers with the standard APIs available on the desktop for developing games. Figure 5 highlights some differences between the Dreamcast DirectX components and the desktop ones.
DirectDraw and Direct3D have been heavily optimized for the PowerVR graphics chip to support nearly all the features using the standard APIs from desktop DirectX. Because the hardware on the Dreamcast is fixed, much of the software emulation code in DirectDraw and Direct3D has been removed to reduce memory footprint. Again, your game will be the only application running, so you have to use full-screen exclusive mode:
lpDD->SetCooperativeLevel(hwndMain, DDSCL_EXCLUSIVE | DDSCL_FULLSCREEN);When using DirectDraw, your single game application loads the DirectDraw library, ddraw.dll, and DirectDraw hardware abstraction layer (HAL), ddhal.dll, and DirectDraw calls are passed through these thin libraries to the hardware for execution. When using DirectSound, your application loads dsound.dll and sndcore.dll and DirectSound calls are passed down through these thin layers to the ARM processor sound driver for execution.
Usage of Direct3D under Dreamcast is very similar to that of its PC equivalent. You create the Direct3D device, create and load textures, and then call BeginScene . Inside the loop you set render-states for each mesh and then call one of the DrawPrimitive* functions. When finished you call EndScene and Flip . One notable difference is that there is no Z buffer on the Dreamcast. However, it is still supported by Direct3D for code compatibility purposes. There are some features of the PowerVR chip that have no equivalent support in desktop Direct3D, like support for a modifier volume, which can be used by the hardware to alter various properties of other polygons that fall inside or outside of it. Direct3D exposes some of these in the current release and many will be in future releases. Using these features does create situations where code is Dreamcast specific, but also allows you to make full use of the Dreamcast hardware while still benefiting from desktop-compatible code for a majority of your game. One DirectX component that is completely missing from Dreamcast is Direct3D Retained Mode, mainly due to its size and performance.
Using DirectInput on the Dreamcast is virtually identical to the desktop. You enumerate input devices using EnumDevices and create device objects for each one found. All the currently available devices for the Dreamcast, such as the game pad, wheel, and arcade stick are supported. The main difference on Dreamcast is that buffered data is not supported, so you must use immediate data for all input. This should be done often but cannot be done more than once per frame.
Networking technologies like DirectPlay, WinInet, Winsock, and Remote Access Service (RAS) are all available on the Dreamcast. DirectPlay for Dreamcast allows game developers to provide network-play functionality similar to that already available on the desktop. WinInet opens the door to various Internet services like WebTV that want to target Dreamcast users. As on the desktop, Winsock and RAS provide low-level access to the underlying protocol stack and support for dial-up connections respectively.
Table 3. Windows CE for Dreamcast DirectX Components
DirectX components Main differences on Dreamcast DirectDraw Full-Screen Exclusive Mode Only.
IDirectDrawClipper not supported.
Overlay surfaces not supported.
Direct3D Immediate Mode Dreamcast-specific flags used to expose some PowerVR features.
Frame buffer cleared after each frame.
No Z-buffer on PowevVR
Direct3D Retained Mode Not Supported.
DirectSound No software mixing.
DirectMusic Will be fully supported later this year.
DirectInput Buffered data not supported. Must poll for immediate data.
Force feedback not supported.
DirectPlay IPX protocol not supported.
DirectShow No audio or video capture support.
Control of DVD players and VCRs not supported.
No cutlist support.
No property pages or resource manager.
As with other platforms that support Windows CE, Visual C++ can be used to develop for Dreamcast. For the first time, this allows game developers the ability to use the same tools for both PC and console game development. The Windows CE for Dreamcast Toolkit includes the Windows CE Toolkit for Visual C++ 6.0, which provides an SH4 compiler, linker, and debugger. All this is available using the standard Microsoft Visual Studio® IDE (see Figure 3). Here we've created a Visual Studio workspace that includes both the x86 Win32 and SH4 Win32 versions of our game, which both use the same source files. The PC version builds using the include and library files from the DirectX 6.1 SDK and the Dreamcast version uses the ones provided in the Windows CE for Dreamcast Toolkit. Even though a large chunk of console game development is done in C and C++ these days, there is sometimes still the need to get down and dirty with assembly. This can be achieved using the inline assembly support of the compiler or using the stand-alone SH4 assembler provided. The tools run under both Windows 98 and Windows NT.
Figure 3 shows some code from the d3dstrm sample included in the Windows CE for Dreamcast Toolkit. This sample uses DirectShow to render a media stream onto a texture that is then mapped onto a waving flag using Direct3D. This sample will work under both Dreamcast and a PC running DirectX 5.0 or later. Of course, you won't be able to run this under Windows CE unless you have Dreamcast development hardware from Sega, but you can still run it on a PC. If you're interested in getting development hardware, visit http://msdn.microsoft.com/isapi/gomscom.asp?Target=/windowsce/developer/resources/dreamcast.asp. The full source of this sample is available on the Microsoft Journal (MSJ) Web site (http://msdn.microsoft.com/isapi/gomscom.asp?Target=/msj/default.htm). Looking at the sample, it's easy to see where code can be shared across platforms and what things need customization. A major difference under Dreamcast DirectX is that you can render the media stream directly into a DirectDraw surface that's in video memory. Restrictions under desktop DirectX force the media stream to be rendered into system memory first, and then copied over to video memory. The sample checks to see if RENDER_TO_DEVICE_MEMORY is defined to determine whether it can render straight to video memory. When building a Dreamcast Windows CE version of the project, this is set.
D3DSTRM sample builds and runs for both the desktop and Dreamcast
// ********************************************************************
//
// Function:
//
// UpdateStreamSample
//
// Description:
//
// Synchronously render the contents of the given multimedia stream sample
// onto the texture surface. On Windows CE, the stream is rendered directly
// into the video memory texture. Restrictions on locking video memory
// surfaces under Windows 9x prevents us from doing this, on the 9x platform,
// so we render into a system memory texture and reload the contents
// into the video memory texture with each update.
//
// Arguments:
//
// IDirectDrawStreamSample* lpddss - Pointer to a DirectDrawStreamSample
// object to update from.
//
// Return Value:
//
// true on success, false on failure.
//
//******************************************************************************
bool UpdateStreamSample(IDirectDrawStreamSample* lpddss) {
HRESULT hr;
// Update the sample only if the application is active
if (!g_bActive) {
return true;
}
// If we are seeking, update the stream clock
if (g_stOffset) {
g_stTime += g_stOffset;
g_lpmmsStream->Seek(g_stTime);
}
#ifdef RENDER_TO_DEVICE_MEMORY
// Make sure the flip has completed before changing texture contents or
// we risk a render timeout
do {
hr = g_lpddsFront->GetFlipStatus(DDGFS_ISFLIPDONE);
} while (hr == DDERR_WASSTILLDRAWING);
if (ResultFailed(hr, TEXT("IDirectDrawSurface::GetFlipStatus"))) {
return false;
}
#endif // RENDER_TO_DEVICE_MEMORY
// Update the texture with the contents of the video stream
hr = lpddss->Update(0, NULL, NULL, 0);
if (hr != MS_E_NOTRUNNING && FAILED(hr)) {
ResultFailed(hr, TEXT("IDirectDrawStreamSample::Update"));
return false;
}
// If we've reached the end of the stream, reset it
if (hr == MS_S_ENDOFSTREAM) {
if (WaitForSingleObject(g_hEOS, 0) == WAIT_OBJECT_0)
{
hr = g_lpmmsStream->Seek(0);
if (ResultFailed(hr, TEXT("IMultiMediaStream::Seek"))) {
return false;
}
}
hr = lpddss->Update(0, NULL, NULL, 0);
if (hr != MS_E_NOTRUNNING && FAILED(hr)) {
ResultFailed(hr, TEXT("IDirectDrawStreamSample::Update"));
return false;
}
}
#ifndef RENDER_TO_DEVICE_MEMORY
if (g_lpddsStreamTarget && g_bStretchStream) {
hr = g_lpddsSystemMemory->Blt(&g_rectTexture, g_lpddsStreamTarget,
&g_rectStream, DDBLT_WAIT, NULL);
if (ResultFailed(hr, TEXT("IDirectDrawSurface::Blt"))) {
return false;
}
}
// Load the source texture into the destination
hr = g_lpd3dtDeviceMemory->Load(g_lpd3dtSystemMemory);
if (FAILED(hr)) {
if (hr != DDERR_SURFACELOST) {
ResultFailed(hr, TEXT("IDirect3DTexture2::Load"));
return false;
}
}
#endif // !RENDER_TO_DEVICE_MEMORY
return true;}Most code will compile without changes for both Dreamcast and the desktop, but there are situations where you'll have to make changes. For example, since there is no real concept of window attributes like the title bar, we don't specify any text for it:
#ifndef UNDER_CE
hWnd = CreateWindowEx(0, g_szAppName, g_szAppName, WS_VISIBLE, 0, 0,
GetSystemMetrics(SM_CXSCREEN), GetSystemMetrics(SM_CYSCREEN),
NULL, NULL, hInstance, NULL);
#else
hWnd = CreateWindowEx(0, g_szAppName, NULL, 0, 0, 0,
GetSystemMetrics(SM_CXSCREEN), GetSystemMetrics(SM_CYSCREEN),
NULL, NULL, hInstance, NULL);
#endif // !UNDER_CEThe bottom line is there will be differences between code that runs on the PC and code for the Dreamcast, particularly if your game makes heavy use of Windows UI components, or you wish to take advantage of Dreamcast features not available on the desktop. For the most part, however, a large chunk of code will work across platforms.
Conclusion
The Sega Dreamcast console is just one example of a cutting-edge consumer platform that takes advantage of industry standard APIs and tools. Microsoft is working with other partners to develop other consumer devices like Advanced Set-Top Boxes (ASTB) and Internet terminals built around Windows CE technologies. With the latest release of Platform Builder, the DirectX and Windows Media Technologies are now available to all OEMs interested in the ever-expanding consumer device market. For more information on developing for Windows CE devices or creating your own Windows CE-based platform, see http://msdn.microsoft.com/isapi/gomscom.asp?Target=/windowsce/developer/default.asp.
For More Information
- For information on the Windows CE Toolkit for Dreamcast and how to sign up for development hardware from Sega, see http://msdn.microsoft.com/isapi/gomscom.asp?Target=/windowsce/developer/resources/dreamcast.asp.
- Further details about DirectDraw, DirectSound, and DirectShow can be found at the DirectX Web site at http://msdn.microsoft.com/isapi/gomscom.asp?Target=/directx/default.asp.
- Windows Media Technologies can be found at http://msdn.microsoft.com/isapi/gomscom.asp?Target=/ntserver/mediaserv/default.asp.
- If you are interested in how to use alpha channels for image compositing, refer to Porter, T. and T. Duff, "Compositing Digital Images," Proceedings of SIGGRAPH '84 , Computer Graphics 18 (3), July 1984.
--------------------------------------------
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. This document is for informational purposes only.
This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.
© 1999 Microsoft Corporation. All rights reserved.
Microsoft, Win32, Windows, and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
Other product and company names mentioned herein may be the trademarks of their respective owners.
© 1999 Microsoft Corporation. All rights reserved. Terms of use.
