The visitors took the lead through Georgios Samaras after 21 minutes when the Greek striker was on hand to fire into the net after Scott McDonald had hit the post following Barry Robson’s corner.
But 10 minutes later Hoops skipper Gary Caldwell was sent off for felling Jamie Mole in the box as the last man.
Jambos captain Michael Stewart scored from the spot to level the scores.
And in the 76th minute Bouzid headed a David Templeton cross past Artur Boruc to leave the visitors stunned.
Glen Loovens saw a header cleared off the line late on as Celtic pressed for an equaliser.
They now trail SPL leaders Rangers by four points, while Hearts climb back up two places to seventh.