It is of concern every time a fine filter starts collecting dispersed additives. When a fine filter is too fine? It depends on the type of lubricant, and the type and load of additives in it.
Most nominal filtration range from around 30 to 40 microns and only 50% of solid contaminants are trapped. I do not recommend to go less than 1 mircron absolute filtration(Efficiency range at 99% capability of removing solid particulates).