The Bairns went into the match as underdogs but produced a spirited performance throughout the match at Hampden.
Despite falling behind to a Marley Watkins goal seven minutes before the break, Falkirk dominated the second half.
Peter Grant levelled the match with 10 minutes remaining, shortly after Caley Thistle defender Carl Tremarco was sent-off for a foul on Blair Alston.
And just as it looked like the Championship club would win the trophy for the first time since 1957, substitute James Vincent netted the decisive goal from a Thistle counter-attack.
“I have got nothing but praise for the players and I’m absolutely gutted for them and the supporters,” Houston said.
“I thought we dominated the second half and we got the equaliser. If I’m being honest I thought there was only one team going on to win it.”