Leading 1-0 from the first leg, Slaven Bilic’s side failed to match the intensity displayed by the hosts and fell behind on 15 minutes when Fabrizio Miccoli converted from close range to level the tie.
The Hammers’ chances of progression were dealt a significant blow on the stroke of half time when centre-back James Tomkins, who scored the winning goal in the first leg at Upton Park, was controversially sent off for his part in an altercation with Birkirkara defender Nikola Vukanac.
The hosts then failed to make their man advantage count before, in the second period of extra-time, Mauricio Mazzetti’s dismissal for a second bookable offence evened up proceedings.
The two sides could not be separated and it was Vukanac, who was involved in Tomkins’ dismissal, who missed the decisive spot-kick that allowed substitute Diego Poyet to seal the victory as the Hammers converted each of their five spot-kicks to reach the next round.
With Diafra Sakho banned following his red card in the previous round West Ham had very few options in attack and rarely threatened. Mauro Zarate was played as a lone frontman and substitute Modibo Maiga proved to be an unreliable source of goals against the Maltese outfit.
The hosts started lively as they looked to cancel out the first-leg deficit, with James Collins’ indecision almost allowing a dangerous set-piece to sweep through the box before being poked behind by Joey O’Brien.
Despite being on the back foot in the opening 10 minutes, the Hammers had the first telling effort of the game as Mark Noble’s corner was cleared as far as Matt Jarvis, who controlled on his chest before firing a low volley straight at Justin Haber in the Birkirkara goal.
That did not dent the early confidence of the home side and Miccoli was on hand to sweep home at the near post after smart work by strike partner Ellenton Liliu.
As West Ham continued to make heavy weather of the tie, Noble was cautioned after tangling with Miccoli as the Birkirkara players then began looking to rile up the midfielder.
Mazzetti was then booked for a foul on Zarate before the Hammers’ frustrations continued as Tomkins and Vukanac wrestled with one another in the penalty area ahead of a West Ham free-kick – with the Hammers defender sent off and Vukanac booked for the coming together.
Birkirkara midfielder Marcelina had a tame effort towards the end of the half, which came to an end at the right time for West Ham – who needed to regroup after a poor opening 45 minutes.
Little changed after the interval and Hammers coach Bilic responded by introducing Poyet and Maiga in place of captain Kevin Nolan and Morgan Amalfitano.
Jarvis went down under pressure from Vukanac on the hour mark but referee Enea Jorgji was unmoved by the penalty appeals.
The winger had a more substantial claim rejected shortly afterwards as West Ham still could not work Haber.
The half was bereft of chances and it was no surprise when the game drifted into extra time, with Birkirkara looking to hit on the counter-attack despite their numerical advantage.
Mazzetti’s red card in the early stages of the second period of extra-time levelled the numbers, with Maiga finishing well late on only to see his effort chalked off for offside.
That took the game to penalties and Noble, Zarate, Aaron Cresswell, O’Brien and Poyet all converted successfully to set the Hammers up with a third qualifying round clash with Astra Giurgiu of Romania, conquerors of Scottish side Inverness Caledonian Thistle.