Naiyaravoro powered over with two minutes to go, putting his side back in front after they had gone 31-28 down.
Saints had two men fewer as Tom Collins had been red carded for a collision in the air and Cobus Reinach was sin-binned for a deliberate knock-on that led to a penalty try.
But Chris Boyd's side, backed by a loud travelling faithful, kept fighting and after a period of pressure on the Wasps line, Naiyaravoro powered over.
Saints still had two minutes to see out, but they did it, recording a crucial Gallagher Premiership bonus-point win on the road.
Wasps had been dealt a blow before kick-off as skipper Joe Launchbury was forced to withdraw due to illness, meaning Dan Robson and Thomas Young would co-captain the side.
And Saints put them under pressure from the off, enjoying a series of lineout opportunities in the home half, only to see their territory and possession fail to produce points.
Defences were very much on top early on with both sides producing some big hits to stop the opposition in their tracks.
But with Wasps continually landing poor kicks into the hands of the Saints backs, it was only going to be a matter of time before they opened them up.
And so it proved as Rory Hutchinson started a move off before Dan Biggar timed the final pass perfectly for the onrushing Reinach to score.
Biggar easily converted and it was 7-0 to Saints with 15 minutes gone.
Wasps were so close to responding when Thomas Young gathered an overthrown lineout, but the flanker lost the ball as he tried to stretch for the line.
Wasps did get some points on the board soon after, winning a penalty, which was kicked by Jacob Umaga, after a threatening spell on Saints' line.
Saints were soon back on the front foot but they were losing lineouts and, consequently, the chance to put real pressure on their hosts.
And Wasps made them pay with a real sucker punch on the half-hour mark as Malakai Fekitoa made a huge hit and steal and offloaded superbly for Marcus Watson to score out wide.
Umaga converted well from the touchline and his team led 10-7.
Saints continued to venture forward and after Alex Waller made a brilliant break for his team, Ahsee Tuala sent the ball wide for Teimana Harrison to cruise in for the score.
Biggar converted but missed a drop goal attempt soon after as Saints remained just four points ahead at the end of a half they had dominated.
Saints extended their lead just a minute after the restart as Wood charged down Robson's attempted box kick and gathered to score.
Biggar converted to make it 21-10 and Saints were now on the hunt for the bonus-point score.
They thought they had it when they piled over the line but there was no clear grounding so they booted a penalty for an earlier offence to the corner.
Wasps then held up another big shove, but they were beaten when Reinach found enough space to get over on the right.
Biggar, who was having a superb overall game, landed the conversion and Saints were 28-10 up with 27 minutes to go.
But Wasps hit back as Gopperth caught them off guard following a lineout deep in Saints territory, and Umaga converted to cut the gap to 11 points with 23 minutes to go.
Saints were suddenly all at sea, especially on the counter, and Wasps exposed them again as full-back Matteo Minozzi finished off a lightning quick break.
Umaga converted and Saints now led by just four points with 20 minutes still remaining.
Things went from bad to worse soon after as Collins was involved in an aerial collision with Umaga, with the Saints wing requiring lengthy medical treatment.
When Collins did finally get up, he was greeted by a red card from referee Matthew Carley.
It meant Saints, who were under siege, had a huge task on their hands with 15 minutes of the match remaining.
And it got even harder when Reinach was sin-binned for a deliberate knock-on, with Carley also opting to award Wasps a penalty try in the process.
Saints were 31-28 down and two men down with fewer than 10 minutes to go, but, backed by a huge away support, they kept battling.
And after Wasps were penned in, Saints won a string of penalties that eventually led to Naiyaravoro using his immense power to score.
Biggar held his nerve to land a kick that put his side four points up with two minutes to go.
'Come on you Saints' was ringing around the Ricoh Arena as the away fans tried to carry their team to victory.
And they managed it as Saints kept hold of the ball to record a memorable success.
Wasps: Minozzi; Kibirige, Fekitoa, Le Bourgeois (Gopperth 49), Watson; Umaga (de Jongh 65), Robson (cc); West, Taylor (Cruse 49), Brookes (Toomaga-Allen 55); Flament; Matthews; Willis, Young (cc), Carr (Vailanu 55).
Saints: Furbank; Tuala (Naiyaravoro 64), Hutchinson, Francis (Proctor 55 (Taylor 73)), Collins; Biggar, Reinach; Waller (cc) (van Wyk 55), Haywood (Matavesi 57), Hill (O Franks 55); Moon (Ratuniyarawa 62), Ribbans; Lawes, Wood (Ludlam 57), Harrison (cc).
Referee: Matthew Carley