12/07/2025
Should AI help you with tax advice? I asked Gemini and Copilot over 100 tax questions to find out.
While my practice serves a number of areas, I am probably most recognized for tax guidance generally and international tax compliance specifically. I've had a few clients fall away and then come back this year to clarify something or other (in two cases because they had notices from the IRS), and when I confronted them about why they'd been away and waited so long to ask, they said they were getting very detailed AI answers in regular search results and assumed those answers were right.
This is an updated version of an old argument for me, as every year come April I have found some mistake or another in someone's return done by a tax program. That's out of dozens of previous year returns completed by someone or something else that I'm asked to review, so I'm not saying they stink or it's common, but the mistakes are there. For individual income taxes, most software is good, even the free online ones. The interviews are quite complete. But particularly when it comes to self-employment and sometimes with heavy investor income, there are rules that you have know how to get into the form itself (or file an additional form) to get a deduction or credit or exempt the income or factor the taxability of it over time because the interview's yes/no answer can't give the whole picture. Yes/no is fine for 98% (or whatever) of people, so they go with that, but it does not apply to 100% of people. I wanted to see if the AI guidance was basically that level of homogenizing the user base, or actual error.
I use Gemini for some research myself. It saves me time in the voluminous IRS documentation when there's something I don't know off the top of my head. But there are what appear on the surface to be a lot of contradictions in that documentation from pamphlet to pamphlet and instruction to instruction if you don't already have a solid foundation on when thing one applies and when thing two applies. That gets more complicated with the number of times rules change, or the income thresholds for where they apply change. Since these AI's are essentially interpreting that documentation, any confusion from what they see first or most often can come along with their answer.
I am not going to offer myself up as a sacrifice by saying one AI is better or worse than the other. They were reasonably close on most topics, so I'll use the averages.
International tax compliance was the worst performer. Over 29% of the answers were either wrong or provided incomplete information. To be fair, most of my questions in this category were relatively sophisticated, though they should be. It's a sophisticated topic. Both engines seemed to have a good handle on newer filing requirements for foreign owned single member LLCs and corporate requirements in general, but both engines also were off the mark about what actually needed to be reported in those filings in at least some cases. Neither seemed to get the nuances of specific tax treaties and the number of times those treaties do and don't apply to single-member LLCs and certain income categories, or how to make income fit into those categories legally instead of just saying woe is me and taking the hit. More importantly, some of those wrong answers were simply because they were quoting IRS tax guidance from documents that had been modified more recently than the quoted document.
Most importantly with the booming number of foreign owned US online retailers, including marketplace sellers, they both generally missed the distinctions of exclusively US sales and product storage vs self-shipping or multiple location storage when incorrectly trying to apply the independent/dependent agent argument. I know that's probably gobbledygook to most people reading this, but that just reinforces my point that if you don't already understand what you're asking, any answer may satisfy you. In this case, that mistake would be the difference between none of your income being taxable and all of your income being taxable, literally.
Not all the misunderstandings had such severe consequences, but even on trivial matters nearly a third of complex inquiries yielding flawed or incomplete results is pretty significant.
Individual income taxes for employed people got a 99% completely right score. Individual income taxes for self-employed people dropped down to 96% completely right on general inquiries and then down to 94% on those with multiple income streams and international or investment income.
Both pretty much aced sales taxes and payroll taxes. Both would have been right 100% of the time on retirement plans including 401(K)s, IRAs, and SEP IRAs, if they had been using the most recent guidance. In about 10% of the more complex questions, they both defaulted to outdated documentation on contribution and distribution requirements and limits. Both were too insistent on what qualifies someone as a business in things like day trading and which income must be qualified as self-employment income. They were technically right on the surface, but did not offer special circumstances that could skew the definition of that income in your favor. Most of the time those special circumstances simply involved filing one additional piece of paper or going to one other screen in your tax software to take advantage of that special circumstance.
If you're a W2 employee with or without retirement and investment accounts, these days you are most likely safe with these tools, both the filing software and AI for general knowledge. If there is an error, it's likely to be very small and in a common area where the IRS is likely to catch it for you and simply make the adjustment. You're not going to lose your shirt. If you're self-employed, one forgotten allowance can easily be more than a qualified advisor's fee, never mind penalties on top in some cases. If you're playing a global game on business income or with foreign accounts, of course it sounds self-serving, but let's just say I'm happy to know I won't be out of a job in that area at least for this tax season.
Gordon
