Reuters previously reported that more than 20,000 staff would be cut, which a source familiar with Friday's announcement confirmed, saying 20 per cent to 25 per cent of the tax-collecting agency's workforce would be targeted.
The start of the layoffs and the civil rights office's elimination were first reported by the Washington Post.
The cuts are part of a major overhaul of the federal workforce that has already cost more than 200,000 workers their jobs. US President Donald Trump has tasked billionaire Elon Musk with leading the reshaping and downsizing of the government.
"The IRS has begun implementing a Reduction in Force (RIF) that will result in staffing cuts across multiple offices and job categories," an internal human resources email sent to all staff said on Friday.
The email said 75 per cent of the agency's civil rights office, previously called the office of diversity, equity and inclusion, will be cut and its remaining employees will move under a separate office.
Trump has passed executive orders aimed at dismantling DEI initiatives that he has labelled discriminatory. Civil rights groups have condemned the actions, saying DEI measures help in addressing historical and generational inequity.
The email said the mass layoffs would take place in phases.
They come amid the busiest time of the year for the IRS, with the filing deadline for most individual tax returns falling on April 15.
