High Performance Computing (HPC) generally refers to the practice of combining computing power in a way that delivers much higher performance than one could get out of a typical desktop computer or workstation in order to solve large problems in science, engineering, or business.
You may have heard of supercomputers (The first famous supercomputer is probably "Deep Blue" - the first computer system to beat a reigning World Chess Master (Garry Kaspirov)), and monster machines from companies like Cray and IBM, that work on some of mankind's biggest problems in science and engineering, origins of the universe, new cancer drugs and even weather forecasting. These are very exotic machines by virtue of both the size and technologies inside them, and as a result they are hugely expensive - with the biggest machines costing tens or even hundreds of millions of dollars to build.
A practical way to understand what high performance computers are is to think about what's in them. HPCs basically contain the same components that you would find in an every day desktop base unit — processor, memory, a storage disk, graphics card and an operating system — just more of them and the ability to transfer the data about the system at very high speed.
Most HPCs today are not single massive computers, but comprise a cluster of smaller powerful computers. Each individual computer in a commonly configured small cluster has between one and four processors, and today's processors typically have from two to four cores.
HPC people often refer to the individual computers in a cluster as nodes. A cluster of interest to a small business could have as few as four nodes, or 16 cores. A common cluster size in many businesses is between 16 and 64 nodes, or from 64 to 256 cores.
The point of having a high performance computer is so that the individual nodes can work together to solve a problem larger than any one computer can easily solve. And, just like people, the nodes need to be able to communicate efficiently with one another.