Excel And High-Performance Computing Part I

Fairly recently, the Microsoft Windows Compute Cluster Server 2003 team publically released their product.  What is “Windows Compute Cluster Server”?  Lots of information available here, but, to borrow a line from their site, Microsoft Windows Compute Cluster Server 2003 is “an integrated, standards-based compute clustering platform built on top of Windows Server 2003 x64 Edition”.  The goal of this work is to “empower users” by allowing them to “easily harness distributed computing resources to solve complex problems”.  Here is a bit more info from someone on the Compute Cluster team:

Some details about Windows Compute Cluster Server 2003

Windows Compute Cluster Server 2003 brings together the power of commodity x64 (64-bit x86) computers, the ease of use and security of Active Directory directory service, and the Windows operating system to provide a secure and affordable high-performance computing (HPC) solution. Windows Compute Cluster Server 2003 can be easily and quickly deployed using standard Windows deployment technologies, and additional compute nodes can be added to the compute cluster by simply plugging in the nodes and connecting them. Windows Compute Cluster Server 2003 includes its job-scheduling, deployment, management capabilities and abilities to exploit high-speed interconnects.

The job scheduler component of allows the jobs—discrete activities to be scheduled to perform on the compute cluster. Jobs can be as simple as a single task or can include multiple tasks. In some situations, tasks are serial, running as single processes; in others, they are parallel, running in multiple, intercommunicating processes. In either case, the ordering of the tasks in a job may be arbitrary or it may be determined by dependencies among the tasks. In addition, jobs and tasks can be targeted to specific nodes within the cluster. Nodes can be reserved exclusively for jobs or tasks or can be shared among jobs and tasks.

The Excel team has been working with the Windows Compute Cluster Server team on a couple of scenarios.  I would like to share one of them today, (and since it was recently part of a keynote demo at TechEd, you get streaming video too).  In a later post, I will talk about a second scenario.

The basic idea is that someone has built a workbook that uses a lot of user-defined functions (UDF) that call off-box to calculate, and each function call is reasonably resource-intensive, so the workbook as a whole is slow to calculate, for potentially a couple of reasons.  First, Excel 2003 (and earlier) calculation is single-threaded, so only one UDF can be called at a time.  Second, if the box actually running the UDFs is near capacity, it can function as a bottleneck.

If the workbook is of a certain type (for example, a value-at-risk calculation), there can be tremendous business value in having faster calculation.  As discussed in an earlier post, the multi-threading work we have done in Excel will help calculation speed a lot, enabling multiple UDFs to be called simultaneously.  However, in the scenario described above, there is also benefit to having more computational power where the UDFs are being calculated.  This is a good scenario for Windows Compute Cluster Server.

To make things a lot faster, what you can do (assuming you have Excel 2007 and a Compute Cluster set up) is to deploy your UDF to the cluster, and then use an XLL to (essentially – I am simplifying a tiny bit – and we will make sample code available at some point) call the UDF on the cluster with the appropriate parameters.  If you turn up the number of threads that Excel uses for calculation (to equal to or more than the number of machines that you have in your cluster), you can, in effect, make a huge number of UDF calls from Excel to the cluster simultaneously, which the cluster will calculate and return to Excel.  For example, if you have a 64-way cluster available, and you tell Excel to use 64+ threads for calculation, you will see the UDF portion of your Excel calculation speed up by a factor of nearly 64.  In certain industries, that’s very exciting … we have seen a lot of interest in this work from the financial services industry for example.

Here is a video from Tech Ed that you can watch to see a demonstration of this scenario.  The Excel – Compute Cluster Server demo starts at around the 01:05:14 mark.  Enjoy.