Recent advances in computing, communication, and data storage have led to an increasing number of large digital libraries publicly available on the Internet. Main problem of content-based video retrieval is inferring semantics from raw video data. Video data play an important role in these libraries. Instead of words, a video retrieval system deals with collections of video records. Therefore, the system is confronted with the problem of video understanding. Because machine understanding of the video data is still an unsolved research problem, text annotations are usually used to describe the content of video data according to the annotator's understanding and the purpose of that video data. Most of proposed systems for video annotation are domain dependent. In addition, in many of these systems, an important feature of video data, temporality, is disregarded. In this paper, we proposed a framework for video temporal annotation. The proposed system uses domain knowledge and a time ontology to perform temporal annotation of input video.