Mark Pugh scoring from their first attack in the 19th minute and former Stags' striker Glynn Hurst sealing victory with a second half strike.
The Shakers should have been tested as early as the third minute when Richie Barker raced clear of the defence, but he sliced his shot well wide.
The Stags then went close again when Boulding's acrobatic overhead kick flew past the post.
With the Bury goal under siege, Brian Barry-Murphy twice made vital tackles on the edge of the area as Boulding shaped to shoot.
Those blocks and misses proved vital when, from their first breakaway, Bury snatched the lead.
Glynn Hurst raced clear down the left and his cross fell kindly to Andy Bishop, who side-stepped keeper Carl Muggleton, only for John-Baptiste to block his shot, but the rebound fell straight to the unmarked Marc Pugh who slammed in from close range.
The goal gave Bury confidence to briefly pour forward in search of a second goal and Challinor almost doubled the lead with a close range header which was nodded away from under the bar by John-Baptiste.
The Stags, clearly rocked by the surprise goal, finally got back into their stride on the half hour when Matt Hamshaw swapped passes with Simon Brown, burst into the area but then fired tamely straight at the keeper.
Then Brass' brave lunge blocked Brown's goal bound blockbuster. From the following corner Jon Olav Hjelde's header was saved on the line.
As Bury were pinned back, keeper Jones dived full length to tip away Coke's 20 yard curling drive as it headed towards the bottom corner in first half injury time.
The Stags again dominated at the start of the second half, but more shambolic defending gifted Bury a crucial second goal.
The home defence twice failed to clear, but Hurst took full advantage with a tame 12 yard drive which beat Muggleton.
The Stags, so dominant when they beat promotion favourites Swindon just five days earlier, rarely threatened a fight back leaving them with just one win in six games.