My significant other bought a house where he wants his father, us and our 8-month-old daughter to live. Should we move in?

My SO and I have an 8-month-old daughter and he has decided to buy a house, but I gave him one condition, that it would just be us. Although I do not hate his father, I do feel uncomfortable around him. While I was pregnant and went over to him and my SO apartment, he would open the bedroom door without knocking, look at me while I have my whole belly out rubbing oil on it and start a whole conversation with his son. He’s walked in on adult time, and he stared at us and said he didn’t know I was here, he’s sorry he didn’t mean to open the door, why wasn’t the door locked, his dad finally closed the door when my SO yelled at him to get out. Again, before I got pregnant, I was there every day, so I don’t understand why he would just come in without knocking. Then once I gave birth he came to the hospital while it was my Parents’ Day to visit, we had decided to give our family separate days so it would be less overwhelming for me. His father is a cigarette smoker and came into our hospital room no mask and took my newborn daughter from my mother. When it was time to take pictures, his father told everyone to take off their face masks, which is one of the things I told my SO is mandatory for visit, but my SO just smiled at his dad while I gave him a look to tell him “You better get your dad”. To make it all worse when it was time to go, he decided to kiss my daughter’s face goodbye, in the CORNER OF THE LIPS. Was it not common sense to not kiss a newborn and a child that is not yours? Yes, he made me comfortable, but my family also gave me a look as to why I was letting him do all the things I told them not to do, and guess what he came again the next day doing the same thing. I decided to spend some time at my SO apartment with the new baby since he also had a 3 month leave but it was unbearable there, early in the morning my SO could hear his dad leaving for work and would grab our daughter and run over to him to say goodbye even though we were sleeping then hand her back to me after to put her back to sleep. I would be breastfeeding in the room and he would come in, I was breast feeding in the living room and he would sit at the dining table and tell me not to worry about it because he could not see far, while I’m changing my daughter's diaper her would hover around and watch, he would give me unsolicited advice and basic ones like “make sure you don’t leave stuff around she can choke on”, while I’m wiping my daughter down for bed he would open the door to watch, after smoking and working he would ask to hold my daughter, he would put my daughters hand in his mouth and when he would carry her she would start crying, he replied to me that he just needs some alone time with her and she will love him. At that point I don’t know if it was my postpartum but so many alarms were going off in my head, I broke down crying wanting to go back home. 8 months later he has bought a house with a basement, and I was excited we would have our own place for the holidays. But once we got to the signing of the house his dad was there, he was moving in, I’ve specifically told him if his father was there me and my daughter are not moving in. He signed the contract anyways. We are now in a heated argument. He said he will build a wall to separate the floors, this made me even more mad, he bought a house over 500k just for us to have the upstairs and his father whole the basement. I’ve told him I respect his decision of prioritizing his dad instead of our family and decided not to move in. My SO is now threatening me with child custody because I told him he will not take my daughter without me, the past 8 months I have been raising her, while he never took care of her alone, he would always pass our daughter off to his mom or dad when we visited them. As a woman and as a mother I will not put my daughter at any risk leaving her alone with men, I refuse to. His father is separated has 5 other kids, 12 other grandkids and he’s only close to 1.