Worrying about it will certainly make it worse. There's no single answer to your questions because it varies a lot from man to man depending on a lot of things, including his state of mind, his partner, diet, exercise, and general health.
Is your mild ED primarily mental or physical? Doctors would ask if you have morning erections. If you do, then it may be mostly mental.
Not being able to get an erection from visual stimulation alone is not something that "happens" at a particular age. For some men, it might start happening more regularly at a wide range of ages, or states of health, or relationships, or stress, etc. For other men, visual stimulation might be enough to get an erection at any age. Yes, I've seen general statements by experts to the effect that as a man gets older, he needs direct stimulation to get an erection -- which is baloney -- not true for all men in all relationships. Likewise, there are men who have trouble with erections their whole life, but are otherwise normal in terms of health, etc.
If you've noticed a change in recent years, you might want to mention it in your annual checkup. The recent wisdom is that ED can be an early warning sign of
heart disease, allowing you plenty of time to take preventive measures.
I really think this is something you need to talk to a doctor about, if you are really worried and the condition persists. Perhaps it is nothing, and it is only your worry that is the problem. I can't hurt to check it out. But you need someone to look at your personal history, health, and overall situation. You cannot go by averages or specific milestones where you can't get an erection by visual stimulation at a certain age, etc.