Islambek Kuat struck at the death as ten-man Kazakhstan recovered from letting a two-goal lead slip and earned three points against their main rivals for second place in UEFA European Under-21 Championship qualifying Group 10.
Iceland had an away victory in mind after Stanislav Lunin was sent off for a second bookable offence with four minutes remaining, but Saulius Širmelis's side showed real character to prevail thanks to Kuat's powerful shot from the edge of the box on 89 minutes.
Kazakhstan had looked set for an easy win when Lunin's seventh-minute tap-in was followed by a splendid second from Roman Murtazayev five minutes later – the No14 beating a defender out on the left and firing into the top corner from 25 metres.
However, Iceland had a rethink at the interval and tore back into their hosts. Hjörtur Hermannsson set up Holmbert Fridjonsson to reduce the arrears on the hour, before provider turned scorer seven minutes later, finding Stas Pokatilov's net with a powerful low strike after a penalty-area melee.