Right now, the Four Emperors are certainly among the elites of the One Piece world, when speaking of pirates, and from the side of the Government, Imu and the Elders seem to be the ones at the top.