Abstract: The quality of surgeries in captured videos is modeled in a learning network. For this task, a dataset of surgical video is given with a corresponding set of scores that are labeled by reviewers, to learn a model for quality assessment of surgical procedures. A learned model is then used to automatically assess quality of a surgical procedure, which omits the need for professional experts to manually inspect such videos. The quality assessment of surgical procedures can be performed off-line or in real-time as the surgical procedure is being performed. Surgical actions in surgical procedures are also localized in space and time to provide a feedback to the surgeon as to which action can be improved.