Early in my career I noticed that a certain type of person got to sit with the CEO a lot. At Lands' End it was a finance person who sold the forecasting ability of her team to Leadership. At Eddie Bauer there was a guy ... an analyst ... who was darn good at forecasting and always had the ear of Management. My Director eventually brought this individual over to Nordstrom.
I noticed that these individuals were confident. Were their forecasts accurate? Well, they were accurate enough. You get huge gains from making modest changes to forecasting algorithms ... but eventually you have to work terribly hard and you get almost no improvement whatsoever. So the confidence comes from knowing just how much work you have to do to get consistent, solid results.
That confidence comes from trial and error.
If you haven't done the work, it's hard to know how "variable" your results are likely to be. This causes doubt, and doubt leads to the comment you see at the start of this post.
You'll learn that making any guess ... ANY guess ... is better than pondering all of the reasons why your work is likely to be wrong. Those who can, do ... those who can't ... they criticize forecasts.
How do you know when you are doing a good job of forecasting future sales? The biggest thing is this ... people stop nitpicking the details and instead ask thoughtful questions.