Check in outlook if those are from the 'Suggested Contacts' folder. From your description it seems that you have this enabled and is collecting all those contacts for you.

Check also if you have Skype for Business/Lync added in outlook. This could have also similar effect depending on the settings you chose.