I am NOT giving a stress test to every 50 year old male smoker with hypertension and family history of MI who wants to exercise. That would be so many patients and seems like a bad use of resources.

Tell me why I'm wrong. Is it because he has so many risk factors for CHD (age, sex, smoker, hypertension, family history)?

From UTD: "CHD screening tests are generally not recommended for asymptomatic patients, and cardiac stress testing should not be part of a routine annual physical or health screening examination. However, there are other patients in whom we perform stress testing: patients who need reassurance that it is safe for them to be active, in which a stress test can help providers delineate an activity program with specific levels of exercise to achieve; and patients who have an occupation in which high levels of exertion may be routinely required (eg, farmer)." https://www.uptodate.com/contents/screening-for-coronary-heart-disease

"Rare exceptions are patients with multiple risk factors" https://www.uptodate.com/contents/selecting-the-optimal-cardiac-stress-test

Before undergoing vigorous exercise an exercise stress test should be performed in those with cardiac risk factors (smoker, family history, male, etc.)