Imam Baqer (AS) quotes the Holy Prophet (S) as saying:
There is nothing between Muslim and infidel except if Muslim abandons prayer deliberately or does not perform it out of negligence.
Thawab al-A’mal wa Iqab al-A’mal, P. 231
مَا بَیْنَ الْمُسْلِمِ وَ بَیْنَ الْکَافِرِ إِلَّا أَنْ یَتْرُکَ الصَّلَاةَ الْفَرِیضَةَ مُتَعَمِّداً أَوْ یَتَهَاوَنَ بِهَا فَلَا یُصَلِّیَهَا
The hideousness of forsaking prayer is so much greater than many of forbidden acts. Because in other forbidden acts, one commits them out of lust and negligence but he may even be unsatisfied with what he does but can not control his lust and passion. However, when one forsakes prayer he is some how arrogant against God and he has problems with his beliefs.