When you start seriously optimizing, you're always making a tradeoff. At first, you start moving from sloppy naive algorithms to more specific ones. No big problem there. Then you go from the general-case algorithm to a "tuned" one that biases to your specific situation. Maybe you change to data types that your language can optimize better. Then you start playing directly with memory and you have to think about little things like byte alignments, pipelining and cache size, getting more and more specific to the architecture you are running on.

At each step, the nature of the solution gets a little bit more intertwined with the details.

Parallelization becomes a tradeoff you "feel" much earlier, because you're changing things all the way up at the algorithm level.