Question

what is the simplest way to permanently get rid of an unwanted file?

Answer 1

Using a utility program like "Eraser"

Answer 2

If on a Unix system (ie Linux or Mac), open Terminal and type
sudo rm -f [file path and name]

Answer 3

Select the file, press Shift + Del. That should permanently delete it off your windows system. But it can still be recovered using a recovery software (sometimes). :)