Perfect Cell didn't have a difficult time against Goku. He was playing with him.
If you remember, only when Gohan went SSJ2 did Perfect Cell unleash his full power (which had the whole planet trembling). Before that, the Cell Juniors were enough to defeat even Vegeta with ease. Before he had his power level down, on purpose because he wanted to enjoy the battle between him and Goku. He thought that Goku was the best amongst the others and since Cell loved battle, he wanted to enjoy this "best" the more he could. If he went all berserk against him like Brolly did, he probably would have defeated Goku easily.
2nd, Cell took damage from Goku's Kamehameha, true. But in the end he came right back up. I think the fact that they felt some hope after seeing his body almost destroyed, and immediately after that, turning that this same hope into despair was Cell's goal. He had no chance of dying by this attack. So he allowed it to damage him instead of going into full defense/power mode.
Brolly was a brute who used full Strength at anything. His simple SSJ form was weak as hell. Goku was fending him off while not even being an SSJ himself.
Goku defeated Brolly after acquiring the remaining strength of the already beaten up Z Warriors. He, himself was beat up as well. From my point of view, "that" Goku is way weaker than any SSJ2 warrior we ever saw at battle. Excluding maybe adult Gohan who had stopped training and had thus become weaker than his teen self. (Vegeta said so after seeing Gohan battling that Demon King guy who was a servant of Babidi)
Remember in the 2nd Brolly movie, Gohan became SSJ2 for a few seconds and beat up Brolly really bad. However it seemed like he couldn't maintain the form and he collapsed before having a chance to finish Brolly. Then a combined Kamehameha defeated Brolly. Pardon me for saying so, but a Kamehameha of two kids (okay Trunks was using his own blast) and Gohan (along with the one by the imaginary Goku, in the same fashion as when Gohan destroyed Cell) didn't really seem as such power to me. Now keep that in mind.
And by keeping that in mind remember the epic battle of Goku vs Vegeta. Brolly had no chance at all.
I mean yeah, he never trained, and if he did he might have bested them all with ease, on that I agree. But since he never did, his power level remained the same. The Legendary Super Sayian mode than only he could use was pretty awesome since he made a guy who as a SSJ couldn't beat the "simple" Goky able to defeat everyone together without even trying. But that was only then. His power level remained the same. Goku's, Vegeta's, Gohan's etc surpassed him by a huge margin.
At least that's how it seems to me.