There are a number of reasons why an oil can stiffen to a gel but we need more info. First what type of oil is it; mineral oil, hydrotreated, PAO, ester?
With a new system there can be residual rust inhibitors, freeze protection chemicals and the like. Gelling without sludge, odor or hot spots is probably not overheating so I would look for chemical causes like glycol contamination.
Did it smell burnt? Overheating, so do a thermographic scan of ALL parts.
Is the pump noisy? Air bubbles, so check oil, suction line ingress and reservoir returns.
Did it smell bad? Bugs - so do a count.
Dit it feel gritty? Contamination.
Did you test for glycol? Contamination
Dit you test the acid number? Overheating.
Did you test the base number? Contamation.
Are you using the recommended oil?
Ken