I hired my house for residential use. The tenant is selling wine and using the house for even prostitution.

Answer

If you knew that the house would be used for prostitution or selling liquor, then the rental is haraam. Normally it is permissible to rent out one's property to a non-Muslim who will obviously do some haraam acts in the house. But that is not the concern of the owner of the house. He rents it out for residential purposes. But when the house is hired specifically for haraam activities, then it is not permissible and the income is haraam.