Combined with the practical case, the paper analyzes the shortcomings of the evaluation of medical Artificial Intelligence (AI) applications for auxiliary diagnosis and treatment, expounds the building of the evaluation system of medical AI applications for auxiliary diagnosis and treatment in detail from the aspects of the framework, concrete dimensions, evaluation process, etc., points out that the system is helpful to open up the commercial closed loop from technology research and development to application and transformation, promote and perfect the construction of laws and regulations of medical AI applications.