Thousands of IRS employees were laid off last week, leaving many taxpayers wondering if those layoffs will impact them.