The accusations flew hard and fast. He mixes with the wrong crowd. He has friends that are respectable people, to be sure. However, the majority of the crowd which followed Him were the type respectable people would not be seen with in public. It is not as if He can keep these friends hidden away in a different world; some of them follow Him around, and the circle of friends tend to overlap. He recognizes how rich and successful, how presentable someone is, how much 'street-cred' someone has, does not matter: the rich can be very needy, and the poor can be very selfish. Inside, we are all weak and prone to failure; it would be unwise for any of us to cast the first stone. Jesus was a friend of sinners, and we need to be, too. He made Himself of no reputation, and we need to be a little less protective of our own.
Matthew had met Jesus and wanted all his friends to meet the Master as well. What could he do? He could do one thing he really knew how to do well, as a tax collector -- he could throw a big party and invite everybody.
Why can we not do the same -- lay aside time-consuming religious activities, and spend time with people, instead?