Abstract-Organ retraction is a common and important task in both open and laparoscopic surgery to expose the blocked surgical site for the surgeon’s view and operation.  Conventional rigid retractors are held by a surgical assistant, where they often struggle to maintain a proper organ pose and mild force without visibility or any other sensor information during a long intraoperative period. In our research, we propose a vision-based control approach for automatic organ retraction using a compliant robotic instrument to improve the safety and efficacy of this surgical task. To this end, the visual deformation model that simultaneously describes the organ’s pose and deformation is established and the retraction procedure in the typical surgical scenario is formulated afterwards. Feedback control and  motion planning methods are developed to manage the retractor’s position and posture autonomously. The feasibility and accuracy of the proposed strategies were validated experimentally using the state-of-the-art surgical robotic platform, da Vinci Research Kit (dVRK).