Two goals in the first 10 minutes set-up the youthful Casper for only his second win as Bury boss and left Peter Shirtliff with his first major set-back at the Stags.
Shirtliff, like Casper at Gigg Lane, had inspired a mini-revival as Mansfield pulled away from the foot of the table.
He also claimed the scalp of Southampton in the Carling Cup as caretaker manager and then Rotherham in the FA Cup just six days earlier in his first game since being appointed manager.
But his first league game in permanent charge, after three wins from four outings as caretaker boss, turned into disaster.
In the sixth minute Danny Reet's header, after the Stags failed to clear a corner properly, left veteran keeper Kevin Pressman flat-footed.
Four minutes later and the keeper was beaten again as Simon Whaley's deflected 15-yard drive doubled the lead and left the home side shell-shocked.
The unmarked Dwayne Mattis side-footed straight at Pressman as Bury should have secured victory after just 16 minutes.
Mansfield, who never looked likely to repeat the free-scoring of eight goals in their previous two home league games, created half-chances for the recalled Adam Birchall to get a goal back immediately.
Simon Brown also twice snatched at shots from 15 yards and in first-half injury time Richie Barker's close-range finish was scrambled off the line by Colin Woodthorpe when recalled keeper Neil Edwards made a mess of Jake Buxton's hopeful ball into the area.
But Mansfield badly missed the suspended Gareth Jelleyman and Gus Uhlenbeek as Bury worked hard to defend their early lead, led superbly by the experienced David Flitcroft just in front of his back four.
Giles Coke, Adam Rundle and Barker went close to giving Mansfield a lifeline at the start of the second half.
But when Barker was scythed down in the area by Thomas Kennedy in the 72nd minute only for referee Mathieson to wave away long and loud penalty appeals, the Stags knew it was not going to be their night.
And just three minutes later Reet secured victory with a 25-yard screamer, slightly deflected over stranded keeper Pressman.