The paper introduces the research status of traditional knowledge graph in the medical field, analyzes the development situation of medical social media data, elaborates the construction method of multi-modal knowledge graph based on medical social media data, and puts forward that the construction of multi-modal knowledge graph is an inevitable trend, and multi-modal knowledge fusion is helpful to further improve the intelligence of medical expert system.