Soft cores use the configurable logic matrix of the FPGA. You can choose to implement them or not, depending on your use case. They can also be tuned to the use case, adding or modifying CPU instructions, cache structure, e.t.c. This involves writing RTL code, with all the design, verification and backend synthesis work that comes with that. Tools like Synopsys ASIP Designer tries to help with this effort.
Hard cores are not part of the configurable logic matrix but are separate resources on the FPGA. That means they can't be tuned to the use case in the same way as a soft core. The trade-off is that they typically are better optimized with regards to clock frequency and power consumption since the components are made to be a CPU and not generic configurable logic. One example of an FPGA with a hard core CPU would be the Xilinx Zynq devices.
An FPGA is (to a very crude approximation) just a bunch of static RAM organized in an unusual way. If you think of normal static RAM as "address wires go in one side and data wires come out the other", in an FPGA there are no "address wires" -- it's all data in/data out. The memory cells are still just memory cells; what we label the wires is merely a matter of engineering perspective. In a Xilinx memory cell we choose labels for the wires typically used for logic gates.
Anyway in a Xilinx chip the bits of data you put in the memory cells determine what logic function gets executed. That works because in general any particular stored memory -- in any computer, anywhere -- is (conceptually) just a logic function, and conversely all logic is implementable with the stuff we conventionally call memory.
But we typically don't do that, because "real" logic made of fixed-function transistors is much faster than logic built with changeable memory cells. However, there's a market for fully-changeable logic--even if it's slower--and that's what Xilinx chips are.
Every CPU is just a bunch of registers and logic. If you hand me a few million discrete NAND gates, I can use them to build an X86, a RISC-V, and ARM, or whatever. It will be the size of a house and it will be very slow, but it will run the binary code for that processor. With a Xilinx chip, you have a few million NAND gates (or NOR gates or inverters or whatever you like) at your disposal and they're all on one chip and you can wire them up however you want with nothing but software. Bingo: You can build an X86 out of pure logic, and it's all on one chip rather than being the size of a house. That's a soft core.
The nice thing about soft cores is that you can build whatever CPU functions you want and leave off the functions you don't need. If you want to change the design, you just download a bunch of new bits to the Xilinx memory cells. Thus you can change an ARM into an X86 in an instant, without changing any hardware.
Soft cores are very flexible, but they're also slow, because implementing logic with static RAM cells is slower than doing it with dedicated transistors.
That's where hard cores come in: A hard core is a dedicated area of silicon on the Xilinx chip carved out to only implement an ARM chip or a PowerPC or other CPU with fixed-function transistors. So it's fast. The downside is you can't change its functionality on-the-fly. If you decide you'd rather have a PowerPC than an ARM chip you have to change the whole chip.
In both types of cores, you still have a bunch of memory cells left over that you can program to do whatever kind of logic you like.
I've seen numbers like 50 MHz vs 1 GHz, so a factor of 20. But that's just raw clock speed. Here's a very interesting talk by a person named Henry Wong who built an out-of-order instruction-level parallel x86 implementation with a soft core, which compensates somewhat for the much slower clock speed. This is a pretty heroic piece of FPGA wizardry:
