Barker's unstoppable drive into the roof of the net deep into first-half injury time also ended the visitors' five-match unbeaten run.
Swansea had edged the first half, but Mansfield took control in a controversial second half when full-back Gary Monk was red-carded for the third time this season and home winger Simon Brown was also sent off.
Third-placed Swansea had created the first chance when Paul Connor just failed to touch in a Lee Trundle cross.
On a rare home attack Tom Curtis saw his 20-yard drive drift narrowly wide, but it was Swansea who controlled the half.
In the 12th minute Kevin Austin nodded in an Andy Robinson free-kick only for his free header to be ruled out for pushing.
Trundle's low shot on the turn was then held by the diving Kevin Pilkington and Trundle, Robinson and Iriekpen all blasted over when well placed.
As Swansea continued to threaten, Austin's back header brushed the outside of the post.
Three minutes before the break, City keeper Brian Murphy did well to hold Adam Murray's deflected 20-yard shot.
But he had no chance a minute into injury time when Barker's first-time shot flew over his head.
A minute into the second half Derek Asamoah's overhead kick flew over and Mansfield almost paid for that miss when Trundle broke clear in the 55th minute, before Pilkington dived full length to hold his angled drive.
Swansea were again fortunate when Brown raced clear into the area after a mistake by Monk, only for the full-back to make an excellent recovering tackle.
But in the 62nd minute Monk was then sent off after being booked twice in a minute for fouls on Brown.
However, just five minutes later, Brown himself was then red-carded for a headbutt on Robinson as an always niggly game boiled over.