Is there a possible route down from the
toilet that doesn't go through a joist? What I would be thinking is take the
most direct route for the toilet, then run the drains for the sink and tub to
where ever the toilet drain ends up. The tub and sink can use 1.5" pipe,
which is going to be much easier to route than 3".
The other thing I would be thinking about is whether adding a joist and
adjusting the spacing might solve it. In other words, think about what it
would look like if you were building the house new, and about what it would
take to get there from where you are. I just feel that cutting that big a hole
in a joist is a hack thing to do.
On the other hand -- it's your house and you're going to live there.
Cutting the joist is not going to make the house fall down, it might make the ceiling sag or the floor squeak. If it does, you're going to have to fix it. Fixing it might mean cutting open the ceiling downstairs and adding a joist. Whether you do that now or a year from now, it's the same amount of work. So there's something to be said for cutting the hole and taking a wait-and-see attitude about whether more reinforcement is necessary.
Before cutting anything I would have all the fittings laid out and carefully measured to insure that everything fits and you can get the slope you need. A big pipe like that will be fussy about where exactly it wants to run.
What I wouldn't do is a fancy tile job in the new bathroom with it being firmly supported. Tile on a floor that flexes will crack for sure.
i went over last nite doing some sheetrock touchup through the house, filling
nail holes etc, and took another quick look and there is no way to change the
stack location because it's in this 6" wall alongside the hallway, probably built
that way to house the stack, along with heat ducts.
then i looked up in the closet ceiling where i removed some rock, and it even
looks like it'd be a challenge to move the shower drain to where it would
need to be for a tub, puts it in an even tougher place so maybe this isn't
going to work out. i have a contractor coming tomorrow to offer his advice
on the subject, we'll see what he says. certainly have to remove the tiled
shower surround as that does leak, and i would want to waterproof that
entire floor area before i rebuild anyway, with whatever i go with for fixtures.