SharePoint People.aspx Page is Visible for All Users

This is sort of possible. You can do the following to adjust permissions for the items, however, the result may not be exactly what you desire.

Here goes:

On your Search Center site, goto Site Actions > View All Site Content

In the Pages library, set the permissions on the "People" page to only allow the users you want to see it.

In the Tabs In Search Pages and the Tabs In Search Results lists, set the permissions on the "People" item.

This should result in the Tab only showing for the appropriate users and access to the page as well. However, if they are redirected here from somewhere else (i.e. the My Site search box), they will get an ugly Access Denied page.