This can be caused by a variety of factors such as:
If you have a desk job where you are sat down for most of the day then try and have regular breaks. Sitting down for a long period of time causes the testes to become overheated which then reduces sperm count.
If you work in an environment in which you are exposed to potentially harmful chemicals then either ask your employer for a change of duties or make sure that you are fully protected whilst doing so.
Go for loose fitting trousers and boxer shorts instead.
If you are prescribed any medication by your doctor then ask him/her about the effects of this on your fertility.