Patients with untreated spinal cord metastases or metastases close to vital organs (as determined by the principal investigator) are excluded Brain metastases that are symptomatic or requiring corticosteroids (except inhaled); subjects must also be off of enzyme-inducing anticonvulsants for > 4 weeks Patients must have no clinical evidence of leptomeningeal or brain metastasis causing spinal cord compression that are symptomatic or untreated or not stable for >= 3 months (must be documented by imaging) or requiring corticosteroids; subjects on a stable dose of corticosteroids > 1 month or who have been off of corticosteroids for at least 2 weeks can be enrolled with approval of the Cancer Therapy Evaluation Program (CTEP) medical monitor; subjects must also be off of enzyme-inducing anticonvulsants for > 4 weeks Symptomatic or untreated leptomeningeal or brain metastases or spinal cord compression Patients must have no clinical evidence of leptomeningeal or brain metastasis causing spinal cord compression that are symptomatic or untreated or not stable for >= 4 weeks (must be documented by imaging) or requiring corticosteroids; subjects on a stable dose of corticosteroids > 1 month or who have been off of corticosteroids for at least 2 weeks can be enrolled with approval of the Cancer Therapy Evaluation Program (CTEP) medical monitor; subjects must also be off of enzyme-inducing anticonvulsants for > 4 weeks Patients must have a CT or MRI scan of the brain to evaluate for central nervous system (CNS) disease within 42 days prior to sub-study registration; patient must not have leptomeningeal disease, spinal cord compression or brain metastases unless: (1) metastases have been locally treated and have remained clinically controlled and asymptomatic for at least 14 days following treatment and prior to registration, AND (2) patient has no residual neurological dysfunction and has been off corticosteroids for at least 24 hours prior to sub-study registration Patients with known leptomeningeal or brain metastases or spinal cord compression should be excluded from this clinical trial No current symptomatic, untreated, or uncontrolled brain metastases present No symptomatic, untreated, or uncontrolled brain metastases present Symptomatic or untreated leptomeningeal or brain metastases or spinal cord compression Leptomeningeal or brain metastases or metastases causing spinal cord compression that are symptomatic or untreated or not stable for >= 3 months (must be documented by imaging) or requiring corticosteroids; subjects on a stable dose of corticosteroids > 1 month or who have been off of corticosteroids for at least 2 weeks can be enrolled with approval of the Cancer Therapy Evaluation Program (CTEP) medical monitor; subjects must also be off of enzyme-inducing anticonvulsants for > 4 weeks Active brain metastasis or leptomeningeal disease; patients with known brain metastases are allowed if metastases have been treated and there is no magnetic resonance imaging (MRI) evidence of progression for at least 12 weeks after treatment is complete and within 28 days prior to the first dose of nivolumab administration; there must also be no requirement for immunosuppressive doses of systemic corticosteroids (> 10 mg/day prednisone equivalents) for at least 2 weeks prior to study drug administration Patients with definitive leptomeningeal metastases Participants with known brain metastases or leptomeningeal metastases must be excluded unless they qualify for enrollment as described below because of poor prognosis and concerns regarding progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events; participants with brain metastases are permitted if metastases have been treated and there is no magnetic resonance imaging (MRI) evidence of progression for at least 4 weeks or more after treatment is complete and within 4 weeks prior to the first dose of nivolumab administration Patients with untreated brain metastases, spinal cord compression, or evidence of symptomatic brain metastases or leptomeningeal disease as noted on computed tomography (CT) or magnetic resonance imaging (MRI) scans should not be included on this study; screening imaging to rule out brain metastases is not required for screening, but should be performed prior to study enrollment if clinically indicated; patients with treated brain metastases and resolution of any associated symptoms must demonstrate stable post-therapeutic imaging for at least 6 months following therapy prior to starting study drug Patients with untreated brain metastases, spinal cord compression, or evidence of symptomatic brain metastases or leptomeningeal disease as noted on computed tomography (CT) or magnetic resonance imaging (MRI) scans should not be included on this study, since neurologic dysfunction may confound the evaluation of neurologic and other adverse events; patients with treated brain metastases and resolution of any associated symptoms must demonstrate stable post-therapeutic imaging for at least 6 months following therapy prior to starting study drug Patients with active or untreated brain metastases or leptomeningeal metastases are excluded from this clinical trial; NOTE: patients with previously treated brain metastases must have stable neurologic status and magnetic resonance imaging (MRI) imaging following local therapy (surgery or radiation) for at least 4 weeks, with no requirement for immunosuppressive doses of systemic corticosteroids (> 10 mg/day prednisone equivalents) for at least 2 weeks prior to study drug administration (stable low dose dexamethasone allowed at discretion of protocol chair) Have distant metastases DISEASE RELATED CRITERIA: Patients must not have known brain metastases, leptomeningeal carcinomatosis or spinal cord compression unless: (1) metastases have been locally treated (including stereotactic body radiation therapy [SBRT], whole brain radiotherapy [WBRT], and surgical resection) and have remained clinically controlled and asymptomatic for at least 14 days following treatment and prior to registration, AND (2) patient has no residual neurological dysfunction and has been off corticosteroids for at least 24 hours prior to registration Clinical or radiologic evidence of new, untreated, and/or progressive brain metastases prior to registration after induction systemic therapy. Patients with untreated brain metastases, spinal cord compression, or leptomeningeal carcinomatosis should be excluded from this clinical trial; patients whose brain metastases have been treated may participate provided they show radiographic stability (defined as 2 brain images, both of which are obtained after treatment to the brain metastases; these imaging scans should both be obtained at least four weeks apart and show no evidence of intracranial progression); in addition, any neurologic symptoms that developed either as a result of the brain metastases or their treatment must have resolved or be stable either, without the use of steroids, or are stable on a steroid dose of =< 10 mg/day of prednisone or its equivalent (and anti-convulsants) for at least 14 days prior to the start of treatment Patients with untreated brain metastases, spinal cord compression, or evidence of symptomatic brain metastases or leptomeningeal disease as noted on computed tomography (CT) or magnetic resonance imaging (MRI) scans are excluded from this clinical trial; while screening brain MRI is not required, it should be performed if clinically indicated at the discretion of the treating investigator; should patient found to have brain metastasis, treatment of brain metastasis must precede the participation in this study Symptomatic or untreated leptomeningeal or brain metastases or spinal cord compression Patients with symptomatic uncontrolled brain metastases; a scan to confirm the absence of brain metastases is not required; the patient can receive a stable dose of corticosteroids before and during the study as long as these were started at least 4 weeks prior to enrollment; patients with spinal cord compression unless considered to have received definitive treatment for this and evidence of clinically stable disease for 28 days