In a sign that there’s little likelihood of Republicans losing control of the U.S. House of Representatives after the November elections, the latest Bloomberg Businessweek focuses instead on the prospect that House Speaker John Boehner might have a tough time keeping his gavel.

So far, 25 GOP House members have said they’re leaving at year’s end. Like Boehner, many are party-line Republicans who argue Democratic policies are wrongheaded but are frustrated with the obstinacy of Tea Partiers who attack fellow Republicans and equate the give and take of legislating with betrayal. Boehner became speaker in 2011, promising to run a more open House where Democrats and Republicans would battle it out to produce bills, however imperfect, that both sides could live with. Instead he’s spent most of his time trying and often failing to keep his right wing from grinding the institution to a halt.

His supporters have stood with him during bitter intraparty battles over the budget and debt ceiling, and contentious issues like immigration, when conservative Republicans tried to undermine Boehner and take control of the agenda. Some GOP veterans no longer want to fight so hard to get so little done. “You have a lot of moderate members looking at things and wondering where their role is in this process,” says Tom Davis, a former House member from Virginia. “These are some of Boehner’s staunchest allies.”

Once they’re gone, he’ll have a harder time keeping control of his unruly flock. Boehner was narrowly reelected speaker in 2013, with conservatives openly calling for his defeat. They’ve made no secret of their desire to try once again to replace him with a more hard-line leader when he’s up for reelection in January, and see opportunity in the departures of so many long-serving members. “We’re going to have a new speaker,” says Representative Tim Huelskamp of Kansas, one of 12 Republicans who didn’t vote for Boehner for speaker in 2013. “Looks like a whole bunch of folks are leaving who are key to him staying as speaker.”