I have taught statistics for some 30 years at University, and I see a major factor simply being attitude. If you don't take an interested, positive attitude, you're not gonna make any effort to learn even the basics, and certainly not any effort to retain something like the times tables, nor to practice it every once in a while.
We used to play guessing games in the car when going up to the cottage, whereby we would each guess the exact minute that we would arrive at the cottage, once we were fairly close, and knew the number of miles/kilometers left before we got there. I would make it simple by traveling, say, 120 Kilometers per hour, which means one kilometer every 30 seconds. Then I would throw in a curveball, by saying that we can only travel 60 Km. per hour once we get onto our little cottage road, which was the last 13 km.
Another exercise was to calculate the total volume of concrete that I needed for our new 24' x 24' garage pad. And the amount of lumber that I would need, both in terms of the studs, and the plywood. It got real fun when we began to calculate the plywood needed for the roof, taking into account the overhang. Then we had to calculate the number of strings of Christmas lights to circumnavigate the fascia of the garage.
Yet another example was to calculate the amount of interest on our mortgage, both per month and per year.
Then we would look at income tax rates, and how the marginal rate varied as a function of how much money you made. (My daughter was real interested in how we pay half the tax rate on capital gains that we pay on earned income. It made her more interested in investing in real estate when she grows up.)
Then, we have 15% total sales tax here in Ontario, and I would get her to calculate the amount but she had to add on in order to pay for something that she wanted to buy.
There are lots and lots of examples that you can use to keep kids' minds active mathematically, but it is a real pain to get them to do these exercises, simply because they don't find them interesting.
Actually, the one time I really got my daughter's attention was when I was submitting my tax return one time, and she saw how much income tax I had to give to the government. She almost had a heart attack at the ripe old age of about 11 years old!