One other thing you should consider is rifle weight. If you're actually going to hunt with the rifles in your collection, I'd say that portability is almost as important as any caliber selection above 416. Some would say above 375.
If you're going on a free range Buffalo or Elephant hunt, you will be walking miles and miles each day, for sometimes several days in a row. Once you get above 40 caliber, rifle weight necessarily must go up. I don't know about you, but I'd much prefer carrying a 8-9 lb scoped 375 or a 9-10 lb scoped 416, than a 12-14 lb anything. It's not that some of the larger bores can't be made lighter, but a full power 458 Lott or 500 Jeffery will need to be quite a bit heavier to comfortably shoot them well.
Also consider that bullet selection is more critical than caliber selection, once you're at or above 375 caliber. With the correct bullet, every legal DG caliber offers enough penetration to kill any and all game, including elephant. With the 375 and 416, you need not have to put up with as much recoil and rifle weight to accomplish the same outcome.
But, with all that said, if you don't mind carrying the extra weight, you love heavy recoil, and you want a true "stopping rifle", I think the 500 Jeffery is the gun for you!
