The longtime justice said he is “concerned about the future of the court” amid growing calls for reform.