Legislators voted Tuesday to fire the director of South Carolina's Department of Public Safety through the state budget, saying his micromanaging is decimating trooper ranks.

The House voted 76-20 to eliminate the director's position and reduce the agency's funding. If the clause remains in the 2017-18 budget, which is still being debated, it would effectively fire Leroy Smith, who's been at the helm since November 2011.

Smith "has destroyed the morale at the highway patrol," said House Minority Leader Todd Rutherford, D-Columbia, who proposed the budget amendment.

Smith is one of three black Cabinet leaders. Rutherford, who is black, said Smith told the Legislative Black Caucus earlier this year complaints about his leadership were due to racism. A House Oversight panel has been investigating the Department of Public Safety.

"As I delved down into the details and the data, it was clear he is the problem," Rutherford said. "The reality is nobody wants to work for him, so they're quitting. The reality is that the people he hires, he fires. It's a total waste of state resources because we're training these troopers for him to fire them for the silliest of reasons."

A spokesman for Smith didn't immediately respond to questions.

Rep. Russell Ott, D-St. Matthews, said Smith's employment is a decision for the governor.

"Let the governor, who wants that responsibility and power, own it," he said.

But Rep. Mike Pitts, R-Laurens, said the move represents legislators' only recourse. Both former Gov. Nikki Haley, who appointed Smith, and Gov. Henry McMaster, who ascended to the office in January, have rejected calls to fire the Cabinet director, he said.

"The budget process is the only way we have to speak our mind about what's going on," said Pitts, a retired Greenville police officer who leads a Ways and Means panel that writes state law enforcement sections of the budget.