I think Fergie hated Raoila, and the posturing he was getting his client to do.
The memory of Pogba's time at United is slightly wrong, what happened was (as an avid youth team follower):
He came through with the FA Cup winning squad alongside Morrison and Tunnicliffe, and was absolutely fantastic. Looked truly top drawer, think that was 10-11.
Then in 11-12 he got his first team debut early in the season, but was mainly in the reserves. Where, to be honest, he wasn't very good. His level dropped significantly - he wasn't even the best reserve player that season. Fergie spoke highly of him and of giving him chances, but at that age he wasn't doing it for the ressies, so I think that pissed Fergie off.
When he left at the end of that season, Fergie said he'd known Pogba and his agent had been speaking behind United's back to Juve for months. It might be revisionist, but that's what let to Fergie being pissed off enough to probably never play him again. That and his wage demands, which apparently were higher than some established first teamers.
Basically, agent does right thing for client, screws club he's at to make loads more money. Hardly surprising in this day and age.