Chris Lucketti had a goalbound effort blocked by Graham Alexander on 20 before Inigo Idiakez headed Mario Licka’s corner over.
On the stoke of half-time, Brian Jensen parried well to keep out Andrew Surman’s 18-yard volley and Burnley raced up the other end to open the scoring.
Goalkeeper Richard Wright turned Chris McCann’s header behind and, from Robbie Blake’s corner, Caldwell headed home from six yards.
The home side were unfortunate not to equalise within 70 seconds of the restart as Bradley Wright-Phillips hit the post with a rasping 12-yard drive.
Jensen tipped Idiakez’s 74-minute strike round the post and then kept out Youssef Safri’s last-gasp effort to ensure Burnley recorded victory.