Ryan McCay was allowed to run at the Hearts defence before unleashing a powerful shot from 20 yards that beat Steve Banks to open the scoring on 20.
Hearts were level within seven minutes through Eggert Jonsson, though, with the aid of a massive deflection.
Hearts then stormed into the lead four minutes before the break through Gary Glen. Saulius Mikoliunas put Glen through on goal, Mark Howard blocked his first shot but Glen followed up the rebound to find the net.
It was 2-2 when Gary Mason’s dipping volley from 25 yards was out of reach for Banks as the ball crashed into the top corner of the net on 78.
But the hosts were awarded a free-kick on 81 and Kingston found the net from 25 yards.