If the question is about the "Recents" tab in the app's own contact picker --
-- those are extracted, every time, from the actual recently sent / received messages.
As those older messages go away, their addresses will too.
There is nothing to clear -- there is no separate data storage for these.