What Do You Do When Your Springtrap Is at the Door?
Which is the result of the Phantoms that you encounter. They aren't scaring you for helping Purple Guy, they do it because they are still angry and full of wrath. It is only until you unlock all the secret minigames, and get the good ending, to which hypothetically speaking the phantoms would not be there anymore.
