The criteria used to evaluate and select open educational resources and textbooks should be almost the same as for traditional textbooks. The following resources can provide useful information to evaluate open textbooks for possible course adoption.
This open book from Iowa State University provides information on creating and using open textbooks. The five sections in the book are: Getting Started, Copyright, Finding OER, Teaching with OER, and Creating OER.
Altman, William S., Kristen Ericksen, and Judith B. Pena-Shaff. An Inclusive Process for Departmental Textbook Selection. Teaching Of Psychology 33, no. 4, p. 228-231. This article discusses a process for selecting a textbook for lower level Psychology courses, but the questions they ask could be useful when evaluating open textbooks or resources for a class.
This website from Achieve.org provides a rubric to help schools choose open educational resources for their school system. Some of the items in the rubric could be applied to choosing open educational resources and textbooks for a college course.