They go to heaven. XD
The big devils we saw usually turn into devil arms, though that makes me wonder what happened to bosses from DMC4.
Small fry is born from dust or other vessel they use, and what I gathered from the little chat Vergil and Arkham have, can be reborn from dust over and over.
Medium demons, like Sid and similar creatures that are not part of seemingly infinite armies, seem to vanish from human world for good, probably lose their soul to some terrible part of demon world. Or maybe they become part of those armies I mentioned?