You are correct, I just assumed wrong...
The committee members include one current athletic director from each of the four "major" conferences—ACC, Big Ten, Big 12, and SEC—also known as the Power conferences. Other members are former coaches, players, athletic directors, and administrators, plus a retired member of the media.
My apologies to Sunshine Pumper and Dayton, looks like the elites pretty much control this thing!