Not painting the fascia, soffit or bargeboards every few years will lead to the wood rotting, and could even let water into your house or rot the end of the roof trusses and this could be a very expensive repair job!
If you live in an old property that has cast iron guttering and downpipes, paint the inside of the gutter with bitumen to help protect it. If you have plastic gutters, take them down so you can paint behind them, this is also a good time to inspect the top of the fascia. It is also a good time to clean your gutters out.