I have been running for a long time without AV, until my kids starting using the machine.
So yeah, why not, if you know what you are doing. It will make your computer notably faster.
I stopped using anything, but my Windows desktop is 50% work computer and 50% home recording studio, so no real surfing or casual web use. I generally use my MBP for that, or checking my bank account or buying anything
As pointed out above, Windows Defender has a less than stellar reputation.
A good starting point with free software to secure your machine would be Malwarebytes (used regularly for on-demand scanning) plus one of the better free always-on AV products out there, like the free editions of Bitdefender, Avast, or Avira.
I'm not sure how well regarded Spyware Blaster is these days (but in any event it's another on-demand service like MBAM and is not a substitute for a proper AV product).
Do you use an administrator account or one with limited rights for your day to day use of Windows? Getting that right will probably give you a far better starting point that installing lots of things like Spyware Blaster.