Patients must have no other chronic disease that would preclude randomization into a lifestyle intervention trial; such diseases include recent myocardial infarction or unstable angina (in the previous 6 months), chronic hepatitis, rheumatoid disease, renal or hepatic disease/dysfunction and diabetics receiving insulin; or other clinical condition limiting ability to walk (recent leg fracture, significant osteoarthritis, related orthopedic conditions, degenerative neurological conditions, etc.) Patients diagnosed with chronic disease/illness precluding their participation (i.e., diabetics receiving insulin, myocardial infarction or unstable angina within previous 6 months, chronic hepatitis, rheumatoid disease, renal or hepatic disease/dysfunction) Patients with evidence of disease outside of the pelvis, including presence of positive periaortic or inguino-femoral nodes Patients with history of active collagen vascular disease Patients who have gross residual disease or distant metastatic disease Patients with pN+ or > T1 disease or who have not had a visibly complete TURBT Patients with evidence of metastatic disease will be excluded; any evidence of non-contiguous spread beyond the primary site as determined by pre or post-operative magnetic resonance (MR) imaging of brain, pre or post-operative MR imaging of the spine, and post-operative cerebrospinal fluid (CSF) cytology obtained from the lumbar CSF space (the requirement for lumbar CSF examination may be waived if deemed to be medically contraindicated); CSF cytology from a ventriculostomy or permanent ventriculoperitoneal (VP) shunt that reveals the presence of tumor cells is indicative of metastatic disease LVEF by ECHO or MUGA within institutional normal limits unless thought to be disease related; DLCO >= 60% with no symptomatic pulmonary disease unless thought to be disease related REGISTRATION #2 - PRIOR TO CONSOLIDATION CHEMOTHERAPY: Patients must have undergone restaging tests according to the study calendar and determined to have no evidence of disease progression Patients with newly diagnosed, biopsy confirmed, extracranial, non-metastatic Ewing sarcoma or primitive neuroectodermal tumor (PNET) of bone or soft tissue are eligible for this study; note:* For the purpose of this study, chest wall tumors with ipsilateral pleural effusions, ipsilateral positive pleural fluid cytology or ipsilateral pleural based secondary tumor nodules will be considered localized disease* Patients with regional node involvement, based on clinical suspicion confirmed by pathologic documentation are considered to be non-metastatic* Patients with discontinuous osseous lesions within the same bone are considered to be non-metastatic* Tumors arising in the bony skull (extra-dural) are considered to be extracranial Patients must have no evidence of metastatic disease; metastatic disease:* Are lesions which are discontinuous from the primary tumor, are not regional lymph nodes and do not share a body cavity with the primary tumor; if there is any doubt whether lesions are metastatic, a biopsy of those lesions should be taken* Skeletal lesions in adjacent bones (trans-articular)* Contralateral pleural effusion and contralateral pleural nodules* Distant lymph node involvement* Patients with pulmonary nodules are considered to have metastatic disease if the patient has:** Solitary nodule > 0.5 cm or multiple nodules of > 0.3 cm unless biopsied and negative for Ewing's** Biopsies of solitary nodule =< 0.5 cm or multiple nodules =< 0.3 cm are not required but if performed and positive indicate metastatic disease Patients must have completed 3 cycles of R-CHOP with no evidence of disease progression HCC patients only:* First line (i.e., no prior systemic therapy) or second-line (with prior first-line sorafenib therapy only) advanced HCC * Child Pugh class A or B7 liver disease * Prior chemoembolization, radioembolization, radiofrequency ablation (RFA), or other local ablative therapies are permissible if >= 6 weeks from procedure with evidence of progression or new metastatic disease, if applicable (Phase I) Prior chemoembolization, radioembolization, radiofrequency ablation (RFA) or other local ablative therapies are permissible if >= 6 weeks from procedure with evidence of progression or new metastatic disease, if applicable cN0 or cN1 disease Clinical or radiographic evidence of metastatic disease cNX, cN2, or cN3 disease Patients with metastatic disease by cranial or spinal MRI evaluation or CSF cytology (unless medically contraindicated) are not eligible Leptomeningeal disease Patients must have at least ONE of the following:* Recurrent/progressive disease at any time prior to enrollment - regardless of response to frontline therapy* Refractory disease: persistent sites of disease (after less than a partial response to frontline therapy, following a minimum of 4 cycles of induction therapy) AND patient has never had a relapse/progression* Persistent disease: persistent disease after achieving at least a partial response to frontline therapy after a minimum of 4 cycles of induction therapy and patient has never had a relapse/progression Patients with prior progressive disease who do not meet criteria above, are eligible as long as they have not been off treatment for > 3 months prior to enrollment on NANT 2011-04 Disease confined to the intraperitoneal and retroperitoneal cavity; Note: nodal disease below the diaphragm, implants adherent to the surface of the liver or intrahepatic lesions will not be exclusionary; patients remain eligible if all intrahepatic tumor is debulked or ablated by the time treatment is initiated Metastatic disease outside the intraperitoneal cavity and retroperitoneum, intrahepatic lesions, or pleural effusions; significant ascites precluding catheter placement; exception: intrahepatic lesions removed or planning to be removed during the debulking procedure Patients with a history metastatic melanoma involving the brain will be excluded if they have active disease or have had active disease within the prior six months that was not controlled with surgery or radiotherapy Patients taking steroids for disease control or pain management For the purposes of this study, aggressive multidrug chemotherapy is defined as chemotherapy including 2 or more agents that must include an alkylating agent and a platinum-containing compound; patients must have ONE of the following:* First episode of recurrent disease following completion of aggressive multi-drug frontline therapy* First episode of progressive disease during aggressive multi-drug frontline therapy* Primary resistant/refractory disease (less than partial response by INRC) detected at the conclusion of at least 4 cycles of aggressive multidrug induction chemotherapy on or according to a high-risk neuroblastoma protocol (examples include A3973, ANBL0532, ANBL09P1, etc.) Definitive clinical or radiologic evidence of metastatic disease N2 or N3 disease detected clinically or by imaging Patients with residual macroscopic disease after surgery Clinical or radiographic evidence of metastatic disease; metastatic workup is not required, but is recommended for patients with clinical stage III disease; Note: isolated ipsilateral supraclavicular node involvement is permitted Patients must not have any other serious concurrent disease or have a history of serious organ dysfunction or disease involving the heart, kidney, liver, or other organ system that would put the patient at undue risk of undergoing therapy Patients must not have a history of chronic liver disease (or cirrhosis) Patients with newly diagnosed, unresectable, metastatic, and measurable ASPS who show clinical evidence of disease progression (including history and increasing physical symptoms) will also be eligible; on-study documentation will include a physicians rationale that supports evidence of clinical disease progression (i.e., increasing tumor pain) Patients with no prior therapy are eligible, provided they have metastatic disease that is not curable by surgery Patients must not have evidence of metastatic tumor or other cancer Progression on lenalidomide as part of first line therapy (lenalidomide-refractory disease)* Lenalidomide-refractory disease is defined as disease progression on or progression within 60 days of the last dose of a lenalidomide-based treatment; patients should have received at least 2 cycles of a lenalidomide-based regimen to be evaluable for refractoriness; examples: 1) progression on lenalidomide maintenance therapy after initial induction +/- consolidation; 2) initial response followed by progression on continuous lenalidomide-dexamethasone +/- elotuzumab or daratumumab Pomalidomide naive disease Clinical American Joint Committee on Cancer (AJCC) (AJCC, 7th ed.) II, IIIA or IIIB (with non-operable disease; non-operable disease will be determined by a multi-disciplinary treatment team within 60 days prior to registration; note: for patients who are clearly nonresectable, the case can be determined by the treating radiation oncologist and/or a medical oncologist or pulmonologist * Patients who present with N2 or N3 disease and an undetectable NSCLC primary tumor are eligible* Patients who refuse surgery are eligible* Patients who develop local recurrence after surgery and rendered candidate for definitive concurrent chemoradiation are also eligible* Patients who have received systemic treatment (up to 4 cycles of induction chemotherapy, or up to 6 months of targeted therapy) are eligible Patients with thyroid disease should be excluded unless euthyroid on suppressive or replacement therapy Patients with disease limited to the skin are not eligible, regardless of how wide-spread Have stable metastatic pancreatic cancer after receiving 8-12 doses of FOLFIRINOX (measurable disease is not required) Chest x-ray negative for metastatic disease Progression: patients deemed to have progressive disease based on clinical deterioration after chemoradiation or radiographic progression outside of the radiation field are not eligible; patients deemed to have pseudoprogression are eligible Metastatic breast cancer (MBC) OR metastatic non-small cell lung cancer (NSCLC) OR metastatic adenocarcinoma of the prostate; the sites of allowed metastases are: peripheral lung, central lung, mediastinal/cervical lymph node, liver, spinal/paraspinal, osseous, and abdominal-pelvic* NOTE: after the required number of evaluable patients have been accrued for a given dose level, the accrual for that metastatic location will be temporarily suspended while the safety of that dose level is assessed; a patient can only be entered onto the trial if all of their metastatic locations are open to accrual (e.g. if central lung is temporarily suspended for safety assessment and the patient has a central lung metastases, regardless of other metastases, they cannot enroll until the safety of dose to central lung is determined) Patients must meet at least one of the following criteria:* Disease progression any time after non-steroidal aromatase inhibitor (AI) use in the advanced disease setting* Relapse while on or within =< 12 months of end of adjuvant non-steroidal AI therapy with or without prior endocrine therapy for advanced disease* NOTE: In either setting, treatment with any prior endocrine therapy must be completed >= 2 weeks prior to course 1 day 1 (C1D1) of study treatment with the exception of exemestane which is permitted in the advanced disease setting within =< 4 weeks immediately prior to C1D1; prior adjuvant exemestane is allowed if the disease free interval is > 12 months from the discontinuation of exemestane; prior faslodex, everolimus, palbociclib or other cyclin-dependent kinase (CDK) inhibitor (e.g. ribociclib, abemaciclib) use are allowed and must have been completed >= 2 weeks prior to C1D1; failure to adhere to this washout guideline will result in a protocol violation ARM II ONLY: Nonspecific or no evidence for disease on standard imaging modality ARM III ONLY: Patients must have identifiable metastatic disease on at least 1 clinically indicated imaging modality; if only soft tissue metastasis, one lesion must measure at least 6 mm or greater; patients must have confirmation of prostate cancer prior to 18F-DCFBC imaging* Note: a patient who is eligible for one arm, subsequently may cross-over into a different arm COHORT A: Patients must have progressive disease, defined as the presence of new or growing lesion(s) on radiologic imaging within 14 months of study enrollment and/or new/worsening disease related symptoms within 14 months of study enrollment; (progression according to RECIST v 1.1 criteria is not required) Extent of disease:* Patients with non-metastatic and metastatic disease are eligible* Initially unresectable patients, with or without metastatic disease, are eligible as long as there is a commitment at enrollment to resect the primary tumor No late relapse with completely surgically resectable disease; patients with late relapses (defined as relapse >= 2 years from the date of completion of the last chemotherapy regimen) whose disease is completely surgically resectable are not eligible; patients with late relapses who have unresectable disease are eligible STEP 2 (CROSSOVER ARM FOR PATIENTS WITH PROGRESSIVE DISEASE) Patients must have melanoma that is metastatic and clearly progressive on prior therapy Prior palliative radiation treatment for metastatic disease to be treated on the protocol (including radiopharmaceuticals) No more than two prior therapies for metastatic disease Patients in the phase I portion must have:* Histologically confirmed diagnosis of metastatic, genitourinary solid tumor* Metastatic disease defined as new or progressive lesions on cross-sectional imaging; patients must have at least:** One evaluable site of disease ** Or, appearance of one new bone lesion Progressive cancer (must be considered no evidence of disease or stable) Patients with thromboembolic disease and on anticoagulation Definitive clinical or radiologic evidence of metastatic disease; required imaging studies must have been performed within 90 days prior to randomization Intrinsic lung disease resulting in dyspnea Patients with bulky tumor on imaging are ineligible; treating physicians are encouraged to contact the study chair to request a rapid central imaging review to confirm fulfilment of these eligibility criteria, if they have concernsBulk tumor is defined as:* Tumor with evidence of clinically significant uncal herniation or midline shift* Tumor with diameter of > 5 cm in one dimension on T2/fluid attenuated inversion recovery (FLAIR)* Tumor that in the opinion of the site investigator, shows significant mass effect in either the brain or spine* Multi-focal/ metastatic disease: Note: A second focus of enhancement in a single FLAIR abnormality is permissible and will not exclude the subject** Patients with multi-focal parenchymal disease are ineligible** Patients with leptomeningeal metastatic disease are eligible* Strata B, D and E patients whose tumor has a significant component involving the brainstem or with significant fourth ventricular compression are ineligible EXCLUSION CRITERIA FOR STRATUM C: Patients with bulky tumor on imaging are ineligible; treating physicians are encouraged to contact the study chair to request a rapid central imaging review to confirm fulfilment of these eligibility criteria, if they have concerns; bulky tumor is defined as:* Tumor with any evidence of uncal herniation or midline shift* Tumor with diameter of > 5 cm in one dimension on T2/FLAIR* Tumor that in the opinion of the site investigator, shows significant mass effect* Metastatic disease: Patients with =< 5 separate foci of metastatic disease not causing mass effect on adjacent parenchyma and each measuring less than 0.5 cm in maximum diameter will be eligible for this arm of the study; patients with diffuse linear leptomeningeal spread are not eligible for this arm of the study* Multi-focal disease: Patients with multi-focal parenchymal disease will be eligible for stratum C if the sum of the product of the maximum perpendicular diameters of all measurable non-contiguous lesions is less than 16 cm^2; in such cases, a minimum of 2 and a maximum of 5 target non-contiguous lesions will be selected; the lower size limit of the target lesion(s) should be at least twice the thickness of the slices showing the tumor to decrease the partial volume effect (e.g., 8 mm lesion for a 4 mm slice) Patient MUST have had progressive disease (radiographic or clinical) while on arm 1 single agent nivolumab while registered to A091401 Patients removed from any immunotherapy for reasons other than progressive disease, including arm 1 single agent nivolumab of A091401, are NOT eligible for re-registration Patient must NOT have received prior endocrine therapy for metastatic disease (i.e., must be first-line endocrine therapy for metastatic disease) Patients with M1 disease or a history of M1 disease Prior use of ibrutinib is allowed unless patient has had disease progression while receiving ibrutinib Patient must have chemosensitive disease as defined by at least a partial response to salvage therapy at their latest assessment Patients with CNS disease or other sites of extra-pulmonary metastases at the time of most recent episode of disease recurrence preceding enrollment Definitive clinical or radiographic evidence of distant metastasis or adenopathy below the clavicles Documented disease refractory to at least one PD1/PD-L1 inhibitor, defined as disease progression following at least 2 infusions of the same drug; radiographic disease progression will be documented by the institutional radiologist based on any radiographic evidence (magnetic resonance imaging [MRI], computed tomography [CT], positron emission tomography [PET], or other modalities, etc.) of disease progression on two separate radiographic scans assessment obtained at least 4 weeks apart; this minimum 4-week interval is required to define PD-1 inhibitor resistance based on imaging; alternatively, clinical disease progression may be documented on examination by the treating investigator Patients with recurrent disease or multiple primaries are ineligible Patients must have had one and only one prior regimen of systemic therapy for metastatic disease unless the patient meets the criteria below Patients with metastatic disease by either MRI evaluation (brain and spine) or lumbar CSF cytology are not eligible; patients who are unable to undergo a lumbar puncture for assessment of CSF cytology are ineligible History or current clinical evidence of moderate-to-severe obstructive pulmonary disease or reactive airway diseases (i.e. asthma) requiring therapy Patients disease must not have micropapillary components No uncontrolled gastric ulcer disease (grade 3 gastric ulcer disease) within 28 days of registration Definitive clinical or radiologic evidence of metastatic disease Have recurrent or refractory/unresectable disease for which there is no known curative therapy* Wild type-GIST: patients with recurrent or progressive disease will be eligible; newly diagnosed patients with resectable localized disease will not be eligible; newly diagnosed patients with metastatic disease and newly diagnosed patients with residual tumor following surgical debulking will be eligible* PHEO/PGL: patients with recurrent or progressive disease will be eligible; newly diagnosed patients with PHEO/PGL that is metastatic at diagnosis and/or unresectable will be eligible; patients with PHEO/PGL with localized (non-metastatic), resectable disease will not be eligible* Renal cell cancer associated with HLRCC: patients with localized, resectable HLRCC-associated renal cell cancer will not be eligible; patients with metastatic and/or unresectable HLRCC-associated renal cell cancer will be eligible Patients must have measurable disease at baseline and 3 or fewer discrete, extracranial metastatic disease sites that are technically amenable to stereotactic body radiation therapy (SBRT) or resection (at least one disease site must be amenable to radiation); some examples of what constitutes specific radiation treatment sites defining distinct metastatic disease sites are as follows: a) A lesion in each adrenal gland represents 2 of 3 sites of metastatic disease allowed to be treated on protocol; b) Similarly to NRG study RTOG 0631, disease in 2 contiguous vertebral bodies (with up to 6 cm of paraspinal extension) can represent one site of disease in the spine; non-contiguous lesions in vertebral bodies separated by one vertebral body free of disease should be viewed as 2 sites of treatment; and c) Two lesions in such close proximity to one another that treatment with one isocenter is more accurate and safer in the liver, lungs, or other similar anatomic locations should be viewed as one site of metastatic disease treatment. For de novo stage IV NSCLC patients (patients with metastatic disease at first presentation), primary disease must be treatable with local therapy in the form of SBRT or hypofractionated radiation. If the primary disease is found in the peripheral or central lung parenchyma without nodal disease, for instance, SBRT may be employed at the discretion of the treating institution. If primary disease is more advanced with involvement of the mediastinum (T4 tumor, N1-N3 disease, etc.), these volumes should be technically treatable with hypofractionated radiation; surgery should only be used for metastatic tumors that can be completely resected by lobectomy, segmentectomy, or wide wedge resection. At least one site of metastatic disease or primary disease must be determined by radiation oncologist to be treatable with radiation. Metastatic disease invading the esophagus, stomach, intestines, or mesenteric lymph nodes if not a candidate for surgery for these lesions. Patients with more than 3 discrete locations of extra-cranial metastatic disease after first-line systemic therapy requiring more than 3 radiation/surgery plans to cover these distinct metastatic disease entities. Subjects must have bi-dimensionally measurable disease, defined as at least one lesion that can be accurately measured in at least two planes* This disease must be located primarily in the supratentorial region* Patients with significant disease that is metastatic outside of the supratentorial region are ineligible Patients for who clinical suspicion is present of metastatic disease in the cerebrospinal fluid (CSF) or spine must have magnetic resonance imaging (MRI) of spine and CSF obtained (lumbar puncture or through Ommaya, external ventricular drain [EVD] or shunt) with negative cytology; patients with CSF that is positive for tumor cells or metastatic disease found on MRI are ineligible Definitive clinical or radiologic evidence of metastatic disease Patients with newly diagnosed, unresectable, metastatic and measurable ASPS will also be eligible if they show clinical evidence of disease progression (including history and increasing physical symptoms); on-study documentation will include a physicians rationale that supports evidence of clinical disease progression (i.e., increasing tumor pain) Patients must have a histologically or cytologically confirmed measurable GIST without platelet-derived growth factor receptor, alpha polypeptide (PDGFRA) or KIT proto-oncogene receptor tyrosine kinase (KIT) mutations; GIST may be newly diagnosed or recurrent provided that it meets criteria for progressive or metastatic disease; metastatic disease refers to disease outside the gastrointestinal (GI) tract, not simply a multifocal primary tumor; testing performed by the Laboratory of Pathology, National Cancer Institute (NCI), unless previously conducted by a CLIA/College of American Pathologists (CAP) external laboratory; analysis will include evaluation of 4 exons of KIT (9, 11, 13, 17) and 3 exons of PDGFRA (12, 14, 18) Prior ACT therapy should be completed, and residual disease documented by either radiographic progression or active disease observed on biopsy (i.e. hematologic or solid tumor malignancy must be deemed active by the treating investigator); the investigator may deem that the disease is active on the basis of a pre-treatment biopsy demonstrating viable tumor cells or clinical progression of disease (i.e. RECIST progression is not required) Known metastatic disease Patients must have unresectable or metastatic, histologically confirmed low or intermediate grade (Klimstra Criteria) pancreatic neuroendocrine tumor (PNET) with radiological evidence of disease progression since last treatment No evidence of residual involved lymph node disease or metastatic disease at the time of registration based on clinician assessment of imaging; the treating physician will determine if incidental lesions on imaging require workup to exclude metastatic disease; if based on review of images, the treating physician determines the patient to be stage III, then the patient is eligible No active known autoimmune disease, including colitis, inflammatory bowel disease (i.e. ulcerative colitis or Crohns disease), rheumatoid arthritis, panhypopituitarism, adrenal insufficiency Patients with metastatic disease Patients must not have metastatic disease (i.e., must be M0); patients must not have locally recurrent disease Definitive evidence of metastatic meningioma Presence of leptomeningeal disease Patients with metastatic disease Patients who have multicentric disease Patients must have achieved stable disease, a partial response, or a complete response at their first disease assessment after initiating first-line platinum-based chemotherapy; patients determined to have progressed (in the opinion of the treating physician) at their first disease assessment are not eligible Definitive clinical or radiologic evidence of metastatic disease; if applicable Definitive evidence of multifocal disease Patients must have evidence of disease progression per RECIST 1.1 prior to enrollment or have evidence of cancer-related pain requiring symptom management with narcotic analgesics