As mentioned in my last post, RapidMind generates machine code at runtime. This is similar to just-in-time compilation, but is done at very specific (and controllable) points in an application’s lifetime – typically during application initialization. The responsibility of generating machine code for a specific hardware target belongs to RapidMind’s backends. Each backend includes code generation support for any targets it supports. For example, the OpenGL backend for GPUs generates OpenGL shading language programs corresponding to a user’s computations. Backends like the x86 and Cell backends generate machine code for those architectures using a custom code generation stack, including a backend optimizer, scheduler, register allocator, etc.

The x86 backend is particularly interesting. Our x86 backend targets x86-based CPUs such as those from AMD and Intel. The x86 instruction set is very old, and modern x86 processors provide more of a translation layer from x86 instructions to some underlying architecture-specific instruction set (the microarchitecture of the processor) than a direct implementation of the x86 instructions. Therefore one x86 processor is not like another - microarchitectural differences between vendors and even between different processor generations are vast. This means that a single x86 binary compiled with just one particular microarchitecture in mind may not perform optimally on another microarchitecture. Even though modern CPUs have features like out-of-order scheduling that help “generic” code execute well, we’ve found there is often still plenty of performance to be obtained by performing microarchitecture-specific optimizations.

This problem (or opportunity?) is further compounded by extensions to the x86 instruction set. Starting with extensions like MMX and 3DNow, processor vendors have been providing new instructions only implemented in newer hardware. These instruction set extensions are generally aimed at accelerating particular types of computations, e.g. by providing vector operations that can compute multiple instances of the same operation at once. Today the prevalent family of extensions is the “SSE” (Streaming SIMD Extensions) family. Many different SSE extensions are implemented in hardware shipping today: SSE, SSE2, SSE3, SSSE3, SSE4A, SSE4.1, and SSE4.2. By 2010 processors will be shipping with support for SSE5 (from AMD) and AVX (a new instruction set extension by Intel). These extensions provide a lot of opportunity for improving performance, but code generated targeting a particular extension will not run on older processors that do not support it. Traditional software development thus has to either use some lowest common denominator (e.g. SSE2, which is supported in most processors shipping over the last 4 years or so) or provide many different binaries of the same code and pick one at runtime. These compatibility issues have really hampered adoption of these extensions.

Both of these issues - microarchitectural differences and new instruction sets - are addressed by our backend and code generation design. Since our platform generates code for performance-critical pieces of applications at runtime, we can check to see exactly which CPU we are running on, and generate code optimized for that CPU. Even though we only require SSE2 support to run, we will generate code that makes use of other SSE extensions if they’re available. We schedule instructions very differently on AMD processors than we do on Intel processors, because of differences in how these processors execute code. Taking advantage of the specific microarchitecture we’re on can yield anywhere from a 10% improvement to a doubling in performance!

We use the same mechanism to optimize generated code based on other factors known at runtime, such as the alignment of arrays in memory or knowledge of certain values being constant, but unknown until the application is actually running (e.g. constants read from a data file during application initialization). Unlike a traditional just-in-time (JIT) compiler, our code generation happens at very specific, predictable, and controllable times. We never interpret code, and we don’t have to profile at runtime to find hot spots. Portions of an application not expressed with RapidMind, such as UI or data-handling code, do not undergo this mechanism. Therefore the overhead impact of this runtime work is minimal, and it becomes worthwhile almost immediately.

This concludes my set of articles on why RapidMind can often get not only improved scalability across cores, but also improved performance on a per-core basis than code expressed without RapidMind. I hope you found it interesting! As always, feel free to leave comments if you have any further questions, and I’ll do my best to answer them.

Second, the ISC conference (http://www.supercomp.de/isc08/content/) is going on right now and there have been a number of new hardware announcements. AMD announced a new FireStream card (FireStream 9250), and NVIDIA also announced a new line of GPUs (Tesla 10P). Both of these products are capable of teraflop performance, which is of course great news for people using RapidMind. At the other end of the scale, the largest Cell BE installation in the world, the LANL Roadrunner (http://www.lanl.gov/roadrunner/), broke the petaflop barrier using an actual benchmark, Linpack. Since RapidMind also targets the Cell BE, this is also good news, as it demonstrates clearly the power of this architecture and its ability to scale in large installations.

Getting back to what we did at SIFMA, we demonstrated a 55x speedup on something called a binomial option pricer. I will talk about this at greater length in an upcoming article, but will mention some interesting points here. First, the binomial pricer, unlike our previous results on Monte-Carlo pricers, is very memory-intensive. It’s similar to iterated convolution and explicit PDE solvers. As we also demonstrated great scalability on Barcelona processors and on the FireStream, this application shows how RapidMind can be used to tackle memory-bound applications as well as compute-bound applications such as Monte Carlo. Second, option pricing is considered a “fundamental primitive” in computational finance. In particular, risk evaluation requires a large number of such evaluations and is an important workload used in day-to-day practice by many financial institutions. The speedup factor we have demonstrated has the potential to reduce such calculations from hours to minutes. As in the other application areas that we target, this can potentially transform the workflow practices where these computations are used. If a computation takes hours, you basically have to run it in batch mode, possibly overnight. If it takes minutes, you can run it many times during the day, as part of an interactive workflow, and use up-to-date inputs, enabling completely new ways to do business. This kind of transformation can create incredible new opportunities for our clients. And as demonstrated by the recent hardware announcement I’ve noted above, multi-core and in particular heterogeneous core processors and their deployments are in a definite growth phase, so even better results can be expected in the near future.

A week later I went to back to check the hives–one colony was a little smaller, not quite 3 pounds, so I wanted to make sure the queen was alive and well and reigning over her brood. I took apart the hive boxes, found the queen in one frame and again, I wasn’t stung. Such friendly bees.

But it’s intimidating when you first open the box and see a bee-covered frame.

You take a deep breath and gather your courage because bees can smell the adrenaline coursing through your veins. I was reading about an informal study recently. Some 200 embedded developers noted why they are slow to adopt embedded multi-core technology. (Survey measures readiness to adopt multicore technology.) To them, it must feel like 60 pounds of bees have swarmed their desks.

The problem looks overwhelming but the reigning messages are wrong. We have the thread locking bee, the single-processor-bias bee– that one took a lot of royal jelly to rear into a queen, and I’ve just spotted the familiar lack-of-determinism bee. This is a messy hive.

A New Hive and a New Reign

RapidMind Fixes the Single Processor Bias Problem

The RapidMind parallel programming model is portable to a wide range of parallel hardware architectures, including vector and stream machines, such as GPUs, as well as distributed memory machines, such as the Cell BE. The system provides a strong execution and data abstraction that is simultaneously modular, portable, and efficient.

The RapidMind platform provides a set of backends. Each manages the execution of RapidMind programs on a particular processor. The RapidMind platform manages communication and data flow between the host processor and target device(s). It handles memory transfers and load balancing, leaving you free to focus on high-level programming. The dynamic runtime-compiler and processor support modules compile RapidMind programs optimally for the specific processor in use.

• The GLSL backend executes RapidMind programs on Graphics Processing Units (GPUs).
• The Cell BE backend executes RapidMind programs on the Cell BE Broadband Engine.
• The x86 backend executes RapidMind programs on AMD and Intel processors.
• The Debug backend executes RapidMind programs on the host processor, compiling programs with a C compiler.

RapidMind Has Advanced Debug Support

The Debug backend executes RapidMind Programs on the host processor, compiling the RapidMind Programs with a C++ compiler. Debug information is generated for the compiled programs, allowing them to be debugged in a debugger using techniques such as setting breakpoints, inspecting values and stepping through code. This allows the RapidMind Programs to be debugged, line by line, within a debugger or IDE.

The RapidMind Inspector allows you to view how data in a RapidMind-enabled application is modified as the application is executed. (It is an optional package available in RapidMind Multi-Core Platform Tools.) The RapidMind Inspector provides graphical views that present not just the Program values at a given iteration of the Program but also of the entire data bound to those values. This allows you to inspect the contents of an array bound to an input value or view the contents of an array bound to an output value. Moreover, the RapidMind Inspector allows you to control the execution of the Program so that you can watch how the data is modified from one iteration to the next.

RapidMind Programs Are Deadlock-Free and Deterministic

The main, underlying problem with multi-threading is non-determinism. Multiple threads running simultaneously do not run in lockstep unless you explicitly synchronize them. However, you want to minimize synchronization because it has a negative impact on performance.

Because these threads do not run in lockstep, and because they can access data structures and devices simultaneously (for example, two threads writing to the same memory location), the result is a very difficult class of bugs to find, reproduce, and solve. Even inserting synchronization constructs does not always make things easier — mistakes in explicit synchronization are what lead to deadlocks, where two threads are waiting for each other and thus never continue.

The exact timing of all of this is determined by many factors, thus making it impossible to know that when you ship your well-tested product, it’s not going to break instantly because timing has changed ever so slightly. RapidMind solves this because threading, synchronization and more are handled by the platform such that your application is deadlock-free, race condition free, and deterministic. Programs written with our platform cannot suffer from deadlock, read-write hazards, or synchronization errors. The platform uses a bulk synchronization model that supports a conceptual single thread of control, making debugging straightforward. The structure of the language makes parallelism explicit, however, encouraging the development and use of efficient and scalable parallel algorithms.

RapidMind Allows for Performance Tuning

You can trace important performance events by using the RapidMind platform performance log. This log contains messages generated whenever the platform notices something relevant to performance, such as the inability to perform a certain optimization, or a feature that is being used inefficiently. Most messages are generated during runtime compilation of the program. However, some important messages (that is, transfers to host memory) are generated when preparing to execute a compiled program. The performance log features several different levels of verbosity. See also “How we get improved performance [using RapidMind] on a single core”

So here are the makings of a fully-functioning hive. The bees know what they’re supposed to do because the right framework has been set. Your work is done here.

·         Multi-threaded applications are more difficult to test than single-threaded applications and are hard to debug.

The main, underlying problem with multi-threading is non-determinism. Multiple threads running simultaneously don’t run in lockstep unless you explicitly synchronize them.  However, you want to minimize synchronization because it has a negative impact on performance.

Because these threads don’t run in lockstep, and because they can access data structures and devices simultaneously (for example, two threads writing to the same memory location), the result is a very difficult class of bugs to find, reproduce, and solve.  Even inserting synchronization constructs doesn’t always make things easier — mistakes in explicit synchronization are what lead to deadlocks, where two threads are waiting for each other and thus never continue.

The exact timing of all of this is determined by many factors, thus making it impossible to know that when you ship your well-tested product, it’s not going to break instantly because timing has changed ever so slightly.

RapidMind solves this because threading, synchronization and more are handled by the platform such that your application is deadlock-free, race condition free, and deterministic!

Another concern:

·         Explicit multi-threading doesn’t scale well as the number of cores increases.

Multi-threading’s explicit “threads” suggest that you use task parallelism as a model. But task parallelism doesn’t scale well because if you have only K threads where K is some constant, you’re never going to get a speedup over more than K cores.

The threading model is built around the concept of task, where every task has a separate sequence of control.  RapidMind does not use task parallelism but uses data parallelism. Data parallelism is based on the fact that applications often operate on collections of data, and units of work are often associated with separate elements of such collections. The RapidMind platform uses an SPMD (Single Program, Multiple Data) stream programming model. Using an SPMD data-parallelism model, allows you to work with familiar concepts like functions and arrays but also directly express parallel algorithms.

And a final issue:

·         Explicit multi-threading can’t leverage the use of accelerators.

Accelerators are parallel machines, and multi-threading is a way to express parallelism in your code. However, the OS threading APIs only target the main CPU cores on which the OS and applications themselves are running. Taking advantage of accelerators requires using further, accelerator-specific, APIs. Furthermore, languages such as C, C++, Java, and almost all other programming languages in common use, assume a shared-memory programming model, where all memory is accessible equally by all computational devices. This isn’t the case for GPUs, for example, where the GPU has a separate memory that it accesses, and data must be explicitly transferred between main memory and GPU memory.

RapidMind automatically manages synchronization between the host and the accelerator, so that you don’t have to.

The remedy is to ascend and the delusional thinking goes away immediately.

It’s not the bends, but nitrogen narcosis is dangerous and completely avoidable. And so is multi-threading.

You can manually multi-thread an application to take advantage of multiple cores, but these projects are time consuming and error prone. Multi-threaded applications are more difficult to develop and test than single-threaded applications. Even experienced software developers find multi-threading challenging and multi-threaded applications hard to debug. Did you misread your gage again? Are you really sure you have air left in your tank?

Like a nitrogen-inebriated diver, you need help to ascend. Otherwise you’ll be stuck at the bottom thinking everything is just fine. Meanwhile, the truth is that multi-threading doesn’t scale well as the number of cores increases. And it can’t leverage the use of accelerators. More importantly, it doesn’t address the issue of memory management (a crucial factor in performance).

RapidMind helps you ascend. The RapidMind platform manages memory, does per-core parallelization as well as multi-core parallelization, transparently enables the use of accelerators (if present), and avoids the debugging challenges of multi-threading.

As a developer you work with a single thread of execution. You write code in standard C++ and use your existing skills, tools and processes. The RapidMind platform then parallelizes the application across multiple cores and manages its execution.
When you ascend, suddenly everything is manageable again.

In the next posts, I’ll discuss some of the specific problems with multi-threading and the advantages of using the RapidMind platform.

RapidMind-enabled C++ code is still C++ code. To be precise, RapidMind-enabled code is C++ code which, at runtime, builds RapidMind programs. Let me give you an example. With RapidMind, you might write the following (extremely trivial) program:

Program my_add = RM_BEGIN {
In<Value1f> a, b;
Out<Value1f> c;
c = a + b; 
} RM_END;

In case you’ve never seen RapidMind code before, this snippet declares a RapidMind program (like a C++ function, but more powerful) called “my_add”. This program is defined to take two inputs (a and b), and produce a single output (c), all of which are scalar floating point values. It produces c by adding together the values in a and b, and that’s it. This is all valid C++ code – all the words you see in blue are types or macros provided by the RapidMind libraries and headers. Later on, you can take my_add and apply it to arrays of numbers, and RapidMind will generate machine code and manage its execution (in parallel!) for you.

Those lines of C++ code compile (through your C++ compiler) into a dialogue between your application and RapidMind. If we imagined this being played out on a stage, it might look something like this:

Program my_add

Application: Hey, RapidMind! I want to make a new program! I want to call it my_add.
RapidMind: OK. Gotcha. Here you go, here’s your new program. It’s empty right now.

my_add = RM_BEGIN

Application: And hey, I’d like to define a new body of code, and store it in my_add. OK?
RapidMind: Yup. Let me just hang on to a copy of this my_add program, so I know where to put anything you might want to define next.

In<Value1f> a, b;

Out<Value1f> c;

c = a + b; 

RM_END

for (int i = 0; i < 10; ++i) { c = c + a; }

Of course if you were really serious about the topic of multi-core and many-core development you would read these issues cover to cover, but in this post I’m going to comment on (and recommend) a subset of these articles.

The March/April ACM Queue special issue is specifically about using GPUs for general-purpose computation (something we’ve been doing since at least 1999). The whole issue has excellent coverage of this topic, but I would especially recommend the article entitled “GPUs: A closer look” by Kayvon Fatahalian and Mike Houston. Basically, this article describes the architecture of contemporary GPUs in detail. GPUs have a fairly complex architecture with multiple levels of hardware parallelism, and include multiple cores, massive multithreading, and SIMD tiling, as well as very aggressive mechanisms for latency hiding. While programming platforms like RapidMind do abstract away much of this complexity, the details are interesting and useful to know when performance tuning. For example, the article explains why using too much state in a GPU kernel can decrease performance due to resource oversubscription, and why it’s necessary to give a GPU lots (and lots and lots) of parallelism to work with for maximum performance.

The May special issue of the Proceedings of the IEEE on “Cutting-Edge Computing” also contains a number of useful articles, including one I wrote surveying scalable programming models, including the SPMD programming model used by RapidMind. This article compares and contrasts a number of multi-core architectures (including CPUs, the Cell BE, and both NVIDIA and ATI GPUs), discusses both task and data-parallel programming models, and shows how the SPMD model maps efficiently onto all of these architectures. This issue also includes good articles on the evolution of GPUs, a discussion of mobile GPUs, and a discussion of simulation, recognition, and synthesis workloads.

The final special issue, of IEEE Computer, is not directly targetting multi-core or many-core processors, but does discuss data-intensive workloads such as data mining and recognition tasks. The workloads discussed include both image and string data search, and cover applications in GIS, medical imaging, and spam detection. These articles discuss strategies for parallelizing these workloads on many-core processors, including on the Cell Be and on GPUs.

Although we’ve been involved in this revolution since the beginning, It’s nice to see that the community is starting to take many-core architectures and programming more seriously.  There are tremendous opportunities here, and a fundamental shift in how computers are built and programmed is underway that will lead to vastly improved performance.

                          total performance = scalability across cores × per-core performance

It’s probably no surprise that RapidMind aims to provide excellent scalability across cores. What might surprise you is that we spend significant amounts of effort on single-core performance as well, and RapidMind-enabled apps often outperform code written in C/C++ without RapidMind significantly. Some of our customer case studies show improvements like “25x faster than non-RapidMind-code on 8 cores” – that 25x number is made up of perfect scaling across 8 cores, combined with a 3x performance advantage even on a single core.

How’s that possible? I’m going to explore this in my next few blog posts. Read on for the first reason why we get such good performance:

Reason 1: Our programming model

RapidMind is built entirely inside of C++. When you write RapidMind code, you’re writing C++, nothing else. The code you write, however, is written to use our primitives (”Values, Arrays and Programs”) to express its computational and data-related operations. We’re extremely careful when designing these primitives to avoid making decisions that make it difficult for our platform to optimize your code (while providing you with all the flexibility you need to express your computations, and without adding significant burden on your part).

For example, computations expressed in RapidMind don’t have “plain pointers” like C/C++, where a pointer could (potentially) point to any particular area in memory. Instead, you have pointers to the beginning of any given array (these are called Accessors and ArrayRefs in RapidMind parlance) and you combine these with indices to look up data in an array. This makes it much easier for the platform to determine when two references to memory might refer to the same location (this is known as alias analysis) which in turn makes it much easier for our backends to optimize your computations.

Our programming model is also much more explicit about locality of computations and data than pure C++. This helps us allocate memory efficiently, optimize the right chunks of code together, perform cache optimizations, and much more. Good expression of locality is critical to achieving good scalability, but it’s also an invaluable aid in getting good single-core performance.

By designing our programming model to keep in mind what’s needed to generate efficient code on modern architectures, we are able to get a performance advantage over plain C/C++, which are burdened by language choices that can make it difficult for compilers and processors to do their jobs well. Luckily it turns out a lot of the choices that allow better scalability also allow better single-core performance, and vice versa. Additionally we do this without sacrificing the syntax, modularity concepts (such as functions, classes, and templates) and tools available to C++ developers.

Stay tuned for more reasons we can achieve such high performance per core in my next blog post!

Benchmarks involve comparing the performance of (at least) two things. For example, we may run the same program on two different processors to compare the performance of these processors, or we may compare two different implementations of the same algorithm on the same processor to compare implementation strategies, or we may compare the performance of two different algorithms for solving the same problem.

A benchmark is an experiment. Whenever possible, in order to get meaningful results from an experiment, we should vary one thing at a time, and control the other variables. Unfortunately, when comparing a serial implementation to a parallel implementation, we may have to change the algorithm in order to achieve a parallel implementation, as not all serial algorithms are parallelizable. The other problem is that when moving between processors, an implementation of an algorithm that is optimal for one processor may not be optimal for another. Finally, we often have to change many other things, such as the operating system or the compiler.

Since it may be necessary to change multiple variables when moving from one implementation to another, we have to compare the best possible performance available on either side of a benchmark. It is more reliable to compare the peak performance of tuned implementations than that of untuned implementations.

We use two strategies to achieve peak performance on either side of a benchmark when comparing a RapidMind implementation to a non-RapidMind implementation.

First of all, on the RapidMind side we can use autotuning. RapidMind implementations of algorithms can be parameterized, and these parameters can be chosen using an automatic tuning process to achieve an optimal implementation of that algorithm on a given processor. This is one of the ways that the RapidMind platform supports portable high-performance implementations of algorithms.

Second, we need to seek an independently developed and tuned baseline for any benchmark. For example, in 2006 we did a financial benchmark in cooperation with HP’s High Performance Computing Division. We both started from the same unoptimized baseline code. Then, we did a RapidMind implementation and they tuned the baseline. In the end, after tuning and using appropriate compiler flags with Intel’s icc (an excellent optimizing compiler), the researchers at HPCD achieved over a 3x speedup over the original, unoptimized baseline code. However, the RapidMind implementation was over 32x faster when we ran on an NVIDIA 7800 GPU. Recently, we added x86 CPU support to RapidMind, and find we are still getting a 17x speedup when running against this tuned (single core) baseline on a dual Intel quad-core machine. In other words, RapidMind’s implementation is more than 2x as fast per core on the same processor. In addition, the RapidMind implementation is now over 140x faster than the baseline code when running on newer NVIDIA G80 GPUs.

Since then we have done a number of other benchmarks, using baselines that were either independently developed (for example, in public libraries) or that we have requested independent partners to tune for us. It is important to note that by using tuned baselines, we can be confident that the results of our benchmarks are realistic, and in fact are conservative, speedups.

Of course benchmarks do not tell the entire story, since productivity and portability are also important considerations. However, if performance is the primary goal of moving to multi-core processors and accelerators, then we have to be able to measure it properly.

This is an important question for RapidMind, since we help developers squeeze every drop of performance out of all the processors in their computers. Fortunately, many applications need or want all the performance they can get.

Note the two general categories of performance requirements: “need” and “want.”

Some applications just need performance. They may have an absolute requirement, such as the need to meet a certain frame rate, to respond to real-time events within a certain latency, or to complete a financial analysis in time to close a trade. For these applications, “slow” is equivalent to “broken.”

Then there are the applications just that “want” performance, to a greater or lesser degree. There can be several reasons for this, but often this desire can be expressed as an improvement in efficiency. Obtaining higher performance on the same hardware translates into higher space and power efficiency. This can in turn can translate into performing the same computation on fewer machines or with less power, or to tackling larger or more accurate computations within a fixed space and power budget.

For the data center, a desire for efficiency is easy to understand. For the end consumer, improved power efficiency can mean better battery life for mobile devices, less heat, or a lower carbon footprint. On the other hand, some computations can also be scaled up; bigger databases can be searched, higher resolution images can be processed, deeper (and more accurate) financial models can be built and processed within the trading window.

Towards the softer end of the want/need spectrum are interactive applications, where performance improves the responsiveness of the user interface. Below a certain level, the user interface becomes unusable and non-interactive. Conversely, as performance is improved, new types of interactive applications become feasible. It is now becoming possible, for example, to deploy interactive applications that use real-time physics simulation, machine learning, or computer vision.

To put this in perspective, RapidMind has been able to achieve one or even two orders of magnitude improvements in performance, relative to applications written using traditional approaches on the same hardware. We also target multi-core devices, and processors are going to continue scaling up in performance following Moore’s law by adding more cores.

What can an additional two orders of magnitude do for the efficiency of existing applications, and what new applications will such radical improvements in performance enable? The answer is, improvements in performance will enable completely new kinds of interactive applications to be built.