Leading 3-2 from the first leg of their play-off clash in Glasgow, the visitors fell behind to a goal from skipper Markus Rosenberg who was banned for the first leg.
However, just before the interval Parkhead midfielder Nir Bitton had the ball in the net from a corner only for Serbian referee Milorad Mazic to award a foul to the home side.
It appeared a harsh decision and there was more woe in the 54th minute when Malmo grabbed their second goal with a header from substitute Felipe Carvalho which Hoops defender Dedryck Boyata could only divert over the line.
Celtic manager Ronny Deila had passed up on two chances to qualify for European football's elite club competition last season and yet another failure is set to cost the club upwards of £15million
That will be partially offset by the Europa League into which the Scottish champions will fall, although the Norwegian's reputation will take a dent.
It did not need to be this way.
Former Celtic loan player Jo Inge Berget had made the return game in Sweden more fraught than it ought to have been with the second of his two goals scored in the last seconds at Celtic Park.
And with no goals conceded in their last nine qualifying home games, and having overturned a 2-0 first-leg deficit against Salzburg in the previous qualifier to win 3-2 on aggregate, confidence was high in the Malmo ranks.
Celtic right-back Mikael Lustig failed to recover from a hamstring injury which meant 19-year-old Saidy Janko, signed from Manchester United in the summer, was handed his first start in Europe, while Charlie Mulgrew replaced the dropped Emilio Izaguirre at left-back.
The match started and continued at breakneck pace.
In the 20th minute Celtic defender Virgil van Dijk, linked with a £10million move to Southampton, was booked after kicking the ball away on his way back from a Hoops attack.
Three minutes later there was much more for the Dutchman and the visitors to worry about when Rosenberg leapt the highest to knock in Yotun's corner with his shoulder, with Celtic goalkeeper Craig Gordon unable to get to the ball.
Griffiths then picked up a yellow card along with Vladimir Rodic, after Anton Tinnerholm went to the ground claiming foul play with his knee by the Hoops striker, backed up by television replays.
Moments later came the match's most controversial moment.
Malmo defender Kari Arnason appeared to handle Stuart Armstrong's corner before Bitton turned and hooked the ball into the net - only for referee Mazic to signal an infringement in the home side's favour.
There was hardly time to draw breath.
Seconds before the interval Gordon made a great save from Nikola Djurdjic's drive after the Celtic defence had been opened up and the Scottish champions went in at the break both relieved and frustrated.
Kris Commons replaced Armstrong for the start of the second half while Carvalho came on for Rasmus Bengtsson.
Only a minute had elapsed before Rosenberg fired wide from 14 yards, Celtic defender Boyata then getting himself booked for a foul on Berget.
Gordon made a fine double save, first from Rodic and then from Rosenberg for a corner.
However, the Swedish side's second goal was only seconds away, coming when Rodic's cross was met by the head of Carvalho, with Boyata appearing to get the last touch amid some chaos in the Parkhead defence.
Bitton then entered the book for a foul on Djurdjic and in the 72nd minute he was replaced by Nadir Ciftci, as Deila looked for another way back into the tie, while also protecting his midfielder from a possible second card.
Gary Mackay-Steven then came on for James Forrest, but try as they might Celtic could not get back into the tie and will now have to content themselves with Europa League football again.
TWEET OF THE MATCH
"The officials have made a right mess of that Celtic decision
Probably because there aren't enough of them in Champions League games." - broadcaster Mark Chapman @markchapman
https://twitter.com/markchapman/status/636259580742356993
PLAYER RATINGS
Celtic
Craig Gordon: 8 (out of 10)
Saidy Janko: 7
Dedryck Boyata: 6
Virgil van Dijk: 6
Charlie Mulgrew: 6
Scott Brown: 6
Nir Bitton: 7
James Forrest: 6
Stuart Armstrong: 8
Stefan Johansen: 6
Leigh Griffiths: 6
Subs
Nadir Ciftci: 6
Kris Commons: 5
Gary Mackay-Steven: 5
STAR PLAYER
Craig Gordon: Kept Celtic in the tie with several fine saves and was one of Celtic's few successes on the night.
MOMENT OF THE MATCH
When Nir Bitton's goal, which would have made it 1-1 on the night just before the break, was cancelled out controversially by referee Milorad Mazic.
VIEW FROM THE BENCH
Celtic boss Ronny Deila passed up on two chances last season to qualify for the Champions League and has failed yet again, leaving the Parkhead side to deal with the less-glamorous Europa League while his fellow Norwegian Age Hareide has now qualified Malmo to the group stage of the competition two years running.
MOAN OF THE MATCH
From a Celtic viewpoint it will be the refereeing decision involving Bitton's disallowed goal which ultimately might have cost them upwards of £15million for failure to reach the Champions League group stage, although the home side were deserved winners on the night.
WHO'S UP NEXT
Celtic v St Johnstone (Premiership, Saturday, August 29)
Deila felt his side showed fear in Sweden and could not hide his disappointment at another early exit from Europe's premier club competition.
"Today we performed not even close to what we can do," he told BT Sport Europe.
"We have played well in many, many games this season but today we looked scared and uncomfortable on the ball and very stressed and that's my responsibility.
"So I have to go through this what happened and learn from this
That's so important.
"We have to get better, that's the main thing
Now we were close but when you have a performance like this we don't deserve to get in there."
He added: "These are big games for Celtic
Everybody wants to get through
Everybody wants to play well but we didn't play as a team and looked very uncomfortable on the ball.
"You learn through experiences but this experience was very hard
The opportunity was there today but in the end we didn't deserve it and we are very disappointed.
"If the performance had been fantastic and we lost then you can deal with it but today was a disappointing performance and that is even tougher."
Source : PA
Source: PA