I encountered a problem where all the oil in the hydraulic system turns into dark brown jelly. Anyone has any clue to why it happens. By the way it is a new hydraulic system. The system was flushed and refilled but the same problem occurs.

certain brands of hydraulic oil will gel when subjected to a contaminate with a ph of 11 or higher and the operating temperature is at least 40c. suggest looking at what you are using to flush the system with or if there is contamination getting into the system with a higher than normal ph.

Sounds like you may be pumping oil over a relief or other orifice in your system. I have seen this happen on systems that have been run too hot or have had water introduced. I would suggest having an oil analysis done on the oil to determine water content and acid number. I would also check the operating temp and see if it is abnormally high.

There are a number of reasons why an oil can stiffen to a gel but we need more info. First what type of oil is it; mineral oil, hydrotreated, PAO, ester?

With a new system there can be residual rust inhibitors, freeze protection chemicals and the like. Gelling without sludge, odor or hot spots is probably not overheating so I would look for chemical causes like glycol contamination.

Did it smell burnt? Overheating, so do a thermographic scan of ALL parts.
Is the pump noisy? Air bubbles, so check oil, suction line ingress and reservoir returns.
Did it smell bad? Bugs - so do a count.
Dit it feel gritty? Contamination.
Did you test for glycol? Contamination
Dit you test the acid number? Overheating.
Did you test the base number? Contamation.
Are you using the recommended oil?

Ken