John Elder Robison, author of Look Me In The Eyes, talked about his Asperger’s syndrome going un-diagnosed until adulthood. A great book and in it he talks about the adults being the only ones that interacted with him due to his social skills.
People with autism can be challenging. However, as adults, we can handle it. When behaviors get on our nerves, we need to be the ones that keep them from isolation.
Their behaviors are not a choice to be naughty. They can be organic and they can be learned. They may make comments that are insensitive because they cannot predict how it will be taken.
Those behaviors do not need to be ignored. Depending on the person with autism, they may be thankful that you took the time to explain to them how to be social or why something may be off-putting.

Some problematic behaviors just have to be ignored. However, we need to make sure the person behind those behaviors are also not ignored. You do not know how important you might be to someone without positive relationships. We want people with autism to have friendships. Caring adults may be the only thing keeping them from loneliness.