@WI-2021 The second roadblock is recoil. I don't have any problems shooting the .375's, but I've never shot anything bigger (unless you count some truly vicious 12 ga. turkey loads), so I don't know where my recoil tolerance ends. I'd hate to throw a bunch of money into a big bore and find out I can't shoot it worth a damn.
I am a man of modeat means. I bought a 458 Lott just to see what the recoil felt like. Watched the videos, read the reports, and figured it couldn't be that bad...could it? It wasn't. Maybe try to find someone in your area who has something larger than what you have and test fire. Do some reading how to properly hold and fire a rifle with substantial recoil. Met a guy with a 416 Rigby, he let me shoot it. No coaching, nothing. Holy hell, did it twist my body and hurt me. I have a few larger rifles in the line up now. Hold them properly, work your way up in caliber. You will be ok I'd say or tap out at some point when you reach your limit.