The first arrived from a Tommy Mooney penalty on 26 minutes after a rash challenge by Shane Higgs, who brought down Lee Holmes in the area.
And the Saddlers extended their lead when Anthony Gerrard powered home a header.
Walsall were in control but a Paul Connor goal 16 minutes from time gave Cheltenham a lifeline.
The home side threw men forward and in their best spell of the match came close to snatching an equaliser when excellent work by Adam Connolly created an opening for Michael D’Agostino but the midfielder flashed his volley just wide.
D’Agostino nearly made amends in injury time but a brilliant ball swung in by midfielder David Bird just eluded him and Gillespie at the back post.