Nothing can be deducted only added... If your physics diploma will be deemed relevant then you will get points, if not, then you will get 0.
BTW diploma gets you 15 points, while work experience (if you will be able to get references in correct format) can get up to 15 but it's unlikely. In my case I managed to only get 3 to 5 years (5 points) out of 11 years experience, due to companies going bust, ignoring or simply refusing to provide references in correct format. So my suggestion would be to look for 'skill' that fits your diploma rather than work experience... as those 15 points are granted...
Here's relevant answer you might want to glance over..