The former Olympic gold medallist hammer thrower was one of the key public-facing officials at the organising committee.
Murofushi had been involved with the committee since 2014 and helped design the event schedules for both the Olympics and Paralympics, which were delayed by a year because of the COVID-19 pandemic.
He will take over as commissioner of the Japan Sports Agency, an organisation aiming to promote sport nationwide and which is involved in planning for the Olympics.
"I know that Mr. Murofushi will prove a dependable partner at the helm of Japanese sports administration as we work together toward the success of the Tokyo 2020 Games," said Tokyo 2020 President Yoshiro Mori in a statement.
