Because "point of view" was already in circulation for a very long type and the behaviour of the characters have evolved since then.

Because "point of view" can be referred only to protagonists of the story and not to the minor characters.

Because this term referred to vision only, and it can be the case when we hear narrator's voice and the character's vision.