The correct term is ‘system biology’.  Note the singular form system: not “systems biology”.  This last usage is a solecism that became accepted when it had been repeated often enough, a very example of ‘accumulated wrongs become right’.

System biology may be defined as the study of life using the tools of system theory (not ‘systems theory’).  Ludwig von Bertalanffy’s 1968 masterwork is called General System Theory.  (In some of his later writings, the term “systems theory” did occasionally appear.  I have in my collection some copies of his original typescripts, in which he had written “system theory”, but in the published versions they mysteriously mutated to “systems theory” — evidence of the handiwork of an over-zealous copy editor, perhaps…)

Consider the everyday terms ‘vegetable soup’, ‘ten-foot pole’, ‘train station’; NOT ‘vegetables soup’, ‘ten-feet pole’, ‘trains station’.  In mathematics, one says ‘set theory’, ‘group theory’, ‘number theory’, ‘category theory’, ‘system theory’, etc; NOT ‘sets theory’, ‘groups theory’, ‘numbers theory’, ‘categories theory’, ‘systems theory’, ….  Likewise, in biology, one uses ‘cell biology’, ‘population biology’, ‘system biology’; NOT ‘cells biology’, ‘populations biology’, ‘systems biology’.  One may also note ‘computer science’, ‘plant science’, ‘system science’; NOT ‘computers science’, ‘plants science’, ‘systems science’.

Of course one studies more than one object in each subject!  Indeed, one would say in the possessive ‘theory of sets’, ‘biology of populations’, …; one says ‘theory of systems’ and ‘biology of systems’ for that matter.  But the point is that when the noun of a mathematical object (or indeed any noun) is used as adjective, one does not use the plural form.

Grammatically, a noun-used-as-adjective in a compound is called a noun adjunct (also attributive noun or noun premodifier).  So in ‘system biology’, ‘biology’ is the noun and ‘system’ (singular!) is the noun adjunct.  The rule is that in a noun adjunct, the singular form is used.