Even though the Bhoys took the lead early in the second half, Gordon Strachan's side will finish bottom of Group E after failing to win any of their opening five matches.
Instead Aalborg will line up in the UEFA Cup in the New Year, while Manchester United and Villarreal have qualified from the group and have reached the last 16 of the Champions League with one game still to play.
In a dull first half Celtic wasted the best chances, with Shunsuke Nakamura firing over from the edge of the penalty area and Georgios Samaras twice missing close-range opportunities.
Artur Boruc had to make a smart save to push away Michael Jackobsen's 35th-minute header, while Thomas Enevoldsen blasted a shot over the bar immediately after the break.
However, the SPL leaders took the lead in the 53rd minute when Mark Wilson's cross was headed home by Barry Robson.
That should have been that, but Aaborg grabbed an equaliser with 20 minutes left when Caca's long-range strike took a deflection off Paul Hartley and beat Boruc.
The Danes grabbed the winner when Andreas Johansson's cross was cut out by Glenn Loovens, whose attempted clearance hit Gary Caldwell and bounced into the net.