Well, most regular cleanser has enough surfactants in it to take off sunscreen, but it is advised to double cleanse for a more deep cleansing at night.
And if you are feeling cleansing oil is too rough for your skin, it is either the cleansing oil doesn't has enough slip to it, that you feel you are dragging your skin. Or, you are massaging your skin too hard.
And if cleansing oill still doesn't work for you, you can opt for cleansing balm, milk cleanser or micellar water to take off your sunscreen.