It's absolutely possible to be experiencing pelvic pain due to orgasm. Firstly, you live in your body full time, and you're present for every orgasm you have, so I'm inclined to trust the pattern you're seeing. Secondly, I've known more than one person who enjoys sex but actively avoids orgasms for the exact same reason as you.
After complications from a bladder prolapse surgery, a patient named Grace developed a hematoma, or pooling of blood, and she needed pelvic floor physical therapy to help her pelvic muscles relax after holding such a large mass of blood. Without PFPT, Grace's pelvic pain could get so severe that it was unbearable to sit down, and even after her hematoma disappeared, Grace still got flares: sudden onsets of pain that she says felt "like somebody was literally putting a vise into my vagina."