Should I go see my family?

I live with my boyfriend and we just had a baby. I live in MI 4 hours form my family in Detroit and my Dad is coming to visit form CA for a couple weeks. Well I want to go and visit my family for 2 weeks but he says I shouldn't because he thinks im not going to come back, he thinks Im going to end up moving with my mother. I told him that I am coming back, but the only reason he thinks Im not going to come back is because we live in a little tinny town and we have to drive 45min to the nearest Walmart or mall. and there are more jobs and every thing. Im only considering staying because he to be pissed off for ever and I don't want to deal with his attitude.