Patients must not have experienced a grade or worse immune-related adverse event (irAE) (except asymptomatic nonbullous/nonexfoliative rash) or any unresolved irAE grade , nor have experienced a toxicity that led to permanent discontinuation of prior anti-PD-/PD-L immunotherapy Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Patient has had a prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior >= grade immune-related adverse event (irAE) while receiving immunotherapy, or any unresolved irAE > grade Any history of a prior >= grade immune-related adverse event (irAE) while receiving any previous immunotherapy agent Patient has had a prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Have had any prior Grade ? immune-related adverse event (irAE) while receiving any previous immunotherapy agent, have any unresolved irAE Grade >, or any irAE that led to the permanent discontinuation of prior immunotherapy. Any prior grade ? immune-related adverse event (irAE) while receiving a prior immunotherapy agent, or any unresolved irAE > grade Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Participants with any prior >= grade immune-related adverse event (irAE) which began while receiving immunotherapy Participants with any unresolved immune-related adverse event (irAE) at time of study entry\r\n* Note: Subjects with =< grade thyroiditis and/or hypophysitis are an exception to this criterion and may qualify for the study Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior immune-related adverse event (irAE) >= grade while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Patient has had a prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade . Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any unresolved toxicity (> CTCAE grade ) from previous anti-cancer therapy; any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior grade ? immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved immune related adverse events (irAE) > grade Must not have had a grade >= immune related adverse event (irAE) on nivolumab monotherapy (excluding endocrine toxicity managed with replacement therapy) PHASE I STUDY ELIGIBILITY CRITERIA:\r\nAny prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade ; toxicities of prior therapy, excepting alopecia, should be resolved to less than or equal to grade as per National Cancer Institute (NCI)-CTCAE v.; patients with greater than + proteinuria at entry are ineligible PHASE II STUDY COHORT OVARIAN CANCER ELIGIBILITY CRITERIA (MEDI+O, MEDI+C AND MEDI+O+C):\r\nAny prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade ; toxicities of prior therapy, excepting alopecia, should be resolved to less than or equal to grade as per NCI-CTCAE v.; patients with greater than + proteinuria at entry are ineligible PHASE II STUDY COHORT TRIPLE NEGATIVE BREAST CANCER ELIGIBILITY CRITERIA (MEDI+O ONLY):\r\nAny prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade ; toxicities of prior therapy, excepting alopecia, should be resolved to less than or equal to grade as per NCI-CTCAEv. PHASE II STUDY METASTATIC CASTRATE-RESISTANT PROSTATE CANCER COHORT ELIGIBILITY CRITERIA (MEDI+O ONLY):\r\nAny prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade PHASE II COLORECTAL CANCER COHORT (MEDI+C ONLY):\r\nAny prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade ; toxicities of prior therapy, excepting alopecia, should be resolved to less than or equal to grade as per NCI-CTCAEv.; patients with greater than + proteinuria at entry are ineligible Any prior grade >= immune-related adverse event (irAE) while receiving immunotherapy (including anti-cytotoxic T-lymphocyte antigen [CTLA-] or anti-cluster of differentiation [CD] monoclonal antibody [MAb]) or any unresolved irAE of any grade (controlled irAE endocrinopathies are allowed) Must not have experienced an immune-related adverse event (irAE) where the irAE was the reason for permanent discontinuation of prior immunotherapy in the most recent prior treatment regimen Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Has any prior Grade ? immune-related adverse event (irAE) while receiving immunotherapy, including anti-CTLA- treatment, or any unresolved irAE >Grade . Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade except for endocrine adverse events (AEs) managed with replacement therapy; any other AEs unresolved toxicities grade or more from previous anti-cancer therapy, except alopecia, peripheral neuropathy or non-clinically significant lab abnormalities DOSE ESCALATION COHORT: Any prior ? grade immune-related adverse event (irAE) while receiving immunotherapy, including anti-CTLA- and anti-PD-/PD-L treatment, or any unresolved irAE > grade \r\n* Note: Previous immune-related ocular toxicity of any grade is excluded DOSE EXPANSION COHORT: Any prior ? grade immune-related adverse event (irAE) while receiving immunotherapy, including anti-CTLA- and anti-PD-/PD-L treatment, or any unresolved irAE > grade ; Note: previous immune-related ocular toxicity of any grade is excluded Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Patients must not have any prior grade >= immune-related adverse event (irAE) or any unresolved irAE > grade Patients must not have any prior grade >= immune-related adverse event (irAE) or any unresolved irAE > grade (MEDI RE-TREATMENT) Any prior Grade ? immune-related adverse event (irAE) or any prior corticosteroid-refractory irAE Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior Grade ? immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE >Grade Any prior Grade ? immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE >Grade Any prior Grade ? immune-related adverse event (irAE) or any prior corticosteroid-refractory irAE. Any prior Grade ? irAE while receiving immunotherapy Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any prior grade >= immune-related adverse event (irAE) while receiving any previous immunotherapy agent, or any unresolved irAE > grade Any clinically significant Grade ? immune-related adverse event (irAE) Any grade neurologic or ocular irAE For Arms L (pembrolizumab) and M (nivolumab), history of a prior grade or immune-related adverse event (irAE) or any grade ocular irAE from prior immunotherapy