Are you running under an Administrator account? If you use an ordinary user account (which you really should) then you'll be interrupted every few minutes by
User Account Control dialogs asking you if you want to allow the OS to do something or other, usually offering very little explanation of whatever it actually is.
As a technically-minded user you can probably make sense of this, although it's still irritating, but the average user just gets utterly confused and doesn't know how to answer.
After a period of regular use (of that account) these things start to tail off as permissions are gradually approved - but this just means that you go a few days without one, and then it pops up again when you least expect it. Most people find this even more enraging, as they thought they were now safe.
This ultimately leads to a constant low-level nagging doubt as to whether you'll ever get anything done without interruption, which makes it impossible to feel totally comfortable using the system.
You can turn all this off, if you wish, but then you lose all the security improvements over XP SP2, so there seems little point having the new OS. Furthermore, if you turn UAC off and install software, that software may not work if you later re-enable UAC, or may not work for other users.
That seems to be the major issue people have with it, from what I've heard.