Having won the first leg in Glasgow 2-0, the Hoops found themselves under unexpected pressure when their Icelandic opponents took a shock lead on six minutes through Olafur Karl Finsen.
But in front of 1,000 fans and on a plastic pitch, the Scottish champions composed themselves to take control of the tie – midfielder Nir Bitton’s header on 32 minutes easing the nerves before second-half goals from Charlie Mulgrew, Leigh Griffiths and Stefan Johansen saw them through.
Ronny Deila’s side will now play Azerbaijan side Qarabag in the third qualifier, with the first game at Celtic Park next week.
Stjarnan’s opener came courtesy of sloppy Celtic defending, with Jeppe Hansen skipping past Virgil van Dijk before picking out Finsen, who drilled well past Craig Gordon.
Dedryck Boyata thought he’d levelled on 22 minutes when he bundled home after Johansen’s shot had been spilled only for the offside flag to cut short his celebrations.
But Celtic were level when Bitton met Johansen’s corner from close range and as the visitors assumed control, Stuart Armstrong came close with an effort moments later before Bitton forced a smart save with a 25-yard drive.
They had their reward four minutes after the break when Mulgrew took a pass from captain Scott Brown before firing home low from 20 yards.
After another goal was disallowed for offside on 68 minutes – Bitton this time caught by the flag – Gordon was forced into a rare save at the other end to keep out Finsen’s powerful drive, but with the match stretched Gary Mackay-Steven set up Griffiths (88) to sweep the ball into the net, before Johansen converted a Mulgrew cut-back in added-time to complete the rout.