I haven't really studied Wittgenstein's philosophy of mathematics. I read the entry in the Stanford Encyclopedia of Philosophy on this subject, which I highly recommend.
This does NOT qualify me as anywhere near an expert on this subject, but since I've studied math in my life it was interesting to read, whether I completely agree, or even understand, or not.
My own experience of math has had moments where I felt like I was entering a Platonic realm. But, according to what I read, Wittgenstein will have none of this!
If you're like me, and you've experienced an other worldly sensation sometimes in math -- something unexpected suddenly makes sense, like coming over a hill and seeing an awesome view, a space that you can't help but believe somehow existed before you got there -- then you might not like his philosophy. The Encyclopedia says Wittgenstein insisted that math was invented, not discovered. It is a language game that involves truth.
He famously rejects the Incompleteness Theorem. From what I've gathered so far from Stanford Encyclopedia of Philosophy the argument is something like: mathematical propositions are those propositions that can be proved or disproved in a logical calculus, therefore undecidable propositions are not mathematical as defined by the logical calculus. End of story.
Now, if someone else, who actually knows this part of Wittgenstein, happens to read this and wants to disabuse me of my misapprehensions, I would appreciate greatly.
Why do I bring this up? The reason is what I mentioned in the second paragraph. I can only describe some experiences I had as metaphysical, a step away from religious. Wittgenstein is kind of a downer on all of this. Math is purely invented and 'mathematical objects' don't exist until they are being used in the practice of math. It is interesting to me that someone who wanted so badly to be metaphysical would take such a view. Is it an example of his empirical conscience chiding him? Or is it because he prefers to think about God as real rather than numbers?