Competency-based medical education relies on the use of assessment tools, which can describe and/or measure medical competencies and are supported by validity evidence. The exploration of validity evidence ensures, that the assessment tool not only measures what is intended, but also that the interpretation and decisions made are fair and just. In this review, we have described a contemporary approach to validity used in Kane’s framework. We have also used an example to illuminate, how the exploration of validity evidence can be done in a scientific and systematic manner.