A British philosopher and Anglican Bishop, George Berkeley, decided that matter is an illusion; only 'ideas' are real.
This was maddening to some people, because it goes against common sense.
But Berkeley had an issue. Like many religious thinkers, he did not think that the world as we know it is made up of 'matter' -- atoms, and attributes of matter such as heat. To them, that would mean a godless universe. But that's what philosophers such as Hobbes and Locke seemed to be saying.
So Berkeley created a philosophy called 'Ideaism' which asserts that ideas constitute all existence. This universe of illusion is created by the spirit, God.
In the coffee shops of London and the salons of the upper class, little else was discussed.
So some say it was Berkeley himself who made the joke:
"No mind? Doesn't matter.
No matter? Never mind."
Others attribute it to Dr. Samuel Johnson, a.k.a. 'Dictionary' Johnson, famous for his wit.
Does it matter which said it?