Quick Path Interconnect (QPI)

The first Nehalem architecture processor release was the Core i7 for single socket desktop systems in November 2008.
The Xeon 5500 series followed in April 2009 for 2-way server systems.
Both have 4 physical cores, 3 DDR3 memory channels.
The Core i7 has a single QPI channel and the Xeon 5500 has 2 QPI.
The 2-way Xeon system can be configured with a single IOH or with two IOH as shown below.
With one IOH, both processors connected directly to the IOH.
Each IOH has two QPI channels and 36 PCI-E Gen 2 lanes plus the ESI.
With two IOH, each processor connects directly to one IOH, and the IOH also directly connected to each other.

Notice that there are 1366 pins on the Nehalem processor versus 603 or 771 for the Core 2 based Xeon
processor, and the number on pins on the IOH (now without memory channels) is reduced to a more economical 1295 pins.

The QPI has a bandwidth of 12.8GB/s in each direction simultaneously for a combined
bi-directional bandwidth of 25.6GB/s.
Each PCI-E Gen 2 lane operates at 5Gbit/sec for a net bandwidth of 500MB/s per lane, per direction.
A x4 PCI-E Gen 2 channel is rated 2GB/s per direction, and 4GB/s per direction for the x8 channel.
So while the 36 PCI-E Gen2 lanes on the 5520 IOH are nominally rated for 18GB/s per direction,
the maximum bandwidth per QPI is still 12.8GB/s per direction.
Still the dual IOH system would have a nominal IO bandwidth of 25.6GB/s per direction.
It would very interesting to see what the actual bandwidth, disk and network,
the Xeon 5500 system can sustain is.

The 4-way Nehalem architecture (which might be a Dell R910) scheduled for release in late 2009,
looks something like below.

Each processor socket
has 8 physical cores, hyper-threading (HT or 2 logical processors per core),
16M L3 cache, 4 memory channels, and 4 QPI. The architecture of the 4-way
system has each processor (socket) directly connected to all three other
processor sockets. The remaining QPI connects to an IO hub. Another difference
relative to the AMD Opteron system is that the Intel IO hub has two QPI ports,
where each connects to a processor.

Each QPI full link can be split as two half wide links.
The Nehalem EX with four full QPI can support glue-less 8-way system architecture.
Glue-less means that no other
silicon is necessary to connect the processors together.
Each processor
connects directly to all seven other processors with a half-wide QPI link, and
uses the remaining half wide QPI to connect to the IOH.
The Hyper Transport
Consortium describes this arrangement for a future Opteron system with 4 full
HT links per processor
(HT_General_Overview).
The actual 8-way system architectures described so to date do employ half-wide
links, for both current Opteron systems and forth coming Nehalem EX system.