What you need, first and foremost, is a brain. The shiny box you use to execute an idea is no more relevant to the quality of design than the measure of DaVinci’s ability to draw was governed by the type of pencil he used.
That said, we all have personal preferences and requirements based on our clients’ needs. For me, I’d go Mac every time. I just find them less of a technical hindrance between brain and paper.
That's your personal preference. For me, the applications are all identical. No hinderance at all in the way it works.
I've worked PC and Mac for over 20 years (since 1997), and I can categorically say I have had more issues with Macs than I ever had with a PC.
But hey - whatever works for you.
If you spend as much on a PC as you do on a Mac - you'd have a PC that is twice as good (components wise) and lasts twice as long as a Mac.
This is where I feel people fall down in buying a PC. You go to PC World/Currys and there's so much choice of approx 50 laptops/desktops - and you think, if I buy the most expensive one it will be the best. But it's not always the case. Or if you have a choice between 2 and they are both the same price, you would assume they do the same thing - whereas, one might be geared to portability and the other to be a workhorse desktop replacement, so you go for the ultra skinny one because it looks cool.
But if you go to a Mac store, there are 4 tables set up, and each gets more expensive as you go through them - so there's no denying the more you pay the more you get. And your choice is 4, not 44.
I've always maintained buying a PC is complicated, far more complicated than it should be - there's so much competition.
Apple doesn't have competition, as they are the only manufacturers of their products - so the shop looks clean, shiny, the spotlights over each computer exude excellence.
Back to the the OPs question:
Is there any downsides to using a PC for graphic design?
Picking the right laptop/desktop PC is crucial - and it's not straightforward - but I guarantee you it's cheaper and better than any Mac you can buy. If you buy the right one from the right place at the right time.