U.S. Rep. Bruce Poliquin voted on Thursday to repeal the federal estate tax, joining almost every member in his Republican caucus.

It wasn’t a surprise, as the freshman from Maine’s 2nd District told a conservative group he’d vote to cut the tax during the 2014 campaign. It won’t happen now, since President Barack Obama and Democrats have lined up against it.

But a statement on his vote, Poliquin cited its impact on Maine “family-owned farms, fisheries and wood products companies,” saying it’s not fair for people to grow businesses only to pay “the despised death tax.”

Here’s @RepPoliquin‘s statement on voting to repeal the estate tax. #mepolitics #ME02 pic.twitter.com/1B5E7y1X9K

— Michael Shepherd (@mikeshepherdME) April 16, 2015

But Democrats and their allies jumped on Republicans after the vote, with Americans United for Change, a progressive group, saying the tax break benefits “the Paris Hiltons and Donald Trump Juniors” who don’t need it.

The rhetoric on both sides doesn’t really match the tax’s true impact, but it does only affect a small portion of Americans each year.

The federal estate tax — a tax on the right to transfer property at death — only applies to single people with more than $5.4 milion in assets or married couples with more than $10.9 million, according to the IRS.

The Joint Committee on Taxation says repealing it would increase the federal deficit by $269 billion through 2025, with 5,400 estates expected to pay the tax this year, making up just 0.2 percent of all American deaths.

A lot of those people are very wealthy, but they’re not the people Republicans cite when they discuss repealing the tax. They talk about farms and small businesses. But in 2013, the Tax Policy Center estimated that 120 farms and small businesses would pay the tax that year nationwide.

In Maine, that would make for a small impact: The liberal Maine Center for Economic Policy, examining Gov. Paul LePage’s plan to eliminate the state estate tax — which has a smaller exemption than the federal tax — recently extrapolated federal figures to estimate that three farms and three small businesses would pay the tax each year.

But that’s of little concern to Lance Smith, who runs Smith’s Farm, a large broccoli producer based in Presque Isle, with his cousin. He’s one of handful of Mainers that Poliquin’s office says has been in touch about repealing the estate tax.

Smith, 63, said he has been planning for the federal estate tax for 15 years as he plans to transfer the farming operation to the next generation of Smiths, who want to run it. He said he won’t pass on much money, but the assets tied up in the business could make for a large payday for Uncle Sam.

“In my opinion, they either have to repeal it completely or exclude small- to mid-size businesses where the assets are business assets,” he said.

For Smith, it’s a matter of fairness: Why does he have to pay to transfer the family business after investing in the business and paying income taxes along the way?

That’s the basis of the fight in Congress: While it’s not something that affects many people, Republicans don’t think it’s fair to a few, and that’s enough.

But, as Washington Post fact-checker Glenn Kessler asked recently, “Is that enough to justify throwing out a tax that now mainly affects the very wealthy who are passing on stock portfolios that have never been subject to capital gains taxes?”

Democrats would say no. That’s why the vote isn’t going anywhere.