In this paper, a new desktop orthogonal-type robot with a CAD/CAM-based position/force controller is presented for finishing small metallic molds with curved surface. The robot consists of a three-axis robot whose single one has a high position accuracy of 1 μm, which means that it can perform higher resolutions of position and force, compared to general industrial robots. A thin wood stick tool with a ball-end tip is attached to the z-axis through a force sensor. The control system of the robot is comprised of a force feedback loop, position feedback loop and position feedforward loop. The force feedback loop controls the polishing force consisting of tool contact force and kinetic friction forces. The position feedback loop controls the position in pick feed direction. Further, the position feedforward loop leads the tool tip along a spiral path. We first evaluate the backlash that causes inaccuracy in position at the tip of the abrasive tool, by simply measuring the position and force. Next, a surface following control experiment along a lens mold is conducted, in which the mold has axis-symmetric concave areas. Finally, a LED lens mold is further finished by using the proposed system in order to demonstrate the performance and promise.