Need Advice regarding live-in boyfriend's custody arrangements

Glad to have found this site! Hope to get some good feedback. So here goes:
My boyfriend and I have been living together for 5 years. He has two children with his ex-wife (a son, 18, and a daughter, 12). She disowned her son so he has been living with us for 4 years. His daughter lives with her mom but she is with us a LOT. Their visitation agreement says every other weekend with her dad through the school year then every other week and weekend during the summer. Soon we will have had his daughter at our house for 5 weekends in a row. Her mom will text my boyfriend to ask if he can keep her extra time. He never says no. I understand that he wants his daughter but this is taking up my weekends too!
I don’t mean to sound selfish and yes I knew he had kids when we got together. I have 4 of my own and 2 grandchildren so I get it. BUT it seems that his ex is living a very full life with her boyfriend while I’m helping to raise and pay for their children. I don’t know how to approach my boyfriend about this injustice without coming across mean. Recently I’ve just been living my life and not planning to do things with them as much anymore. I’m tired and should be living a life with him like it started out. (LOTS has changed regarding his children since we met!) HELP! and thank you in advance!!