Rectal tumor at baseline which would be considered to require complete TME No prior chemotherapy or surgery for rectal cancer Recurrent rectal cancer Diagnosis of rectal adenocarcinoma Entire tumor must be in the colon (rectal involvement is an exclusion); surgeon confirmation that entire tumor was located in the colon is required only in cases where it is important to establish if the tumor is a colon versus (vs.) rectal primary History and physical including a digital rectal exam 60 days prior to registration Subjects may not have a history of rectal surgery or lower gastrointestinal bleed No prior therapy for rectal cancer. Patients must have histologically or cytologically confirmed A) low lying (i.e. =< 6 cm from the anal verge) rectal adenocarcinoma eligible for concurrent chemoradiation therapy to rectal tumor, B) if the treatment is palliative in the metastatic setting, no additional requirements for tumor size or nodal involvement is needed; C) if the treatment is in the neoadjuvant setting, the tumor must ALSO be high-risk locally advanced rectal cancer defined as T3-4, N+, and/or at risk for a positive radial margin (as determined by the surgeon) Primary resectable rectal cancer No prior pelvic radiotherapy, chemotherapy, immunotherapy, or other anti-cancer treatment for rectal cancer Previous radiotherapy in the pelvic region (eg, prostate) or previous rectal surgery (eg, total mesorectal excision [TME]) or any investigational treatment for rectal cancer For male subjects, the digital rectal examination must not be suspicious for carcinoma of the prostate. Clinical stage T4 (invasion into rectum or ureters) significantly increases the morbidity of the surgery\r\n* Patients with rectal or ureteral invasion will be considered to have unresectable disease Histologically confirmed diagnosis of colon or rectal adenocarcinoma of any clinical stage, previously untreated with chemotherapy or radiotherapy, with a plan to undergo surgical resection no sooner than 7 days from the projected date of study drug initiation No clinically detectable (MR, endoscopy or digital rectal examination [DRE]) tumor present Prior radiation therapy, chemotherapy or extirpative surgery for rectal cancer Rectal cancer located within 10 cm from the anal verge based on proctoscopy and digital rectal examination (DRE) Rectal cancer staged as T4 by pelvic MRI Severely symptomatic rectal tumors (near-completely obstructing, symptomatic bleeding) Prior history of rectal resection Gross residual disease in the prostate fossa appreciated wither on digital rectal examination (DRE) or on imaging, unless biopsy proven not to contain cancer History of rectal surgery History of rectal fistula Biopsy-proven diagnosis of rectal adenocarcinoma Histologically confirmed invasive well-moderately differentiated rectal adenocarcinoma diagnosed within 90 days prior to enrollment. Prior treatment for rectal cancer. Patients with biopsy-proven adenocarcinoma, stage II- IV rectal cancer. Patients must be candidates for planned surgical resection of their primary rectal cancer 8 - 12 weeks after completion of neoadjuvant chemoRT, even if stage IV. Histologically proven rectal cancer Patient is a candidate for elective rectal resection Surgically unresectable rectal cancer Dukes C colon cancer, high risk Dukes B colon cancer, Dukes B rectal cancer or Dukes C rectal cancer (see Appendix 1 for definition of High Risk Dukes B) Patients with rectal disease Patients with convincing evidence of extraprostatic extension or T4 disease on digital rectal examination (DRE) History of prior invasive rectal malignancy, regardless of disease-free interval Prior significant rectal surgery (hemorrhoidectomy is acceptable) Rectal fissure, fibrosis, stenosis, or other anatomic abnormality precluding insertion of transrectal device Prior surgical procedure involving peri-rectal and peri-prostatic area History of ulcerative colitis or other chronic inflammatory conditions affecting rectum (includes rectal fistula, anal stenosis) Tumors with a lumen sufficient to allow the positioning of the rectal applicator (standard probe/scope) confirmed by the treating physician Tumors of less than 4 cm thickness from the rectal mucosa documented at the time of staging images Previous exposure to chemotherapy for rectal cancer Digital rectal exam within 90 days of registration on study Histologically documented adenocarcinoma of the colon; the gross inferior (caudad) margin of the primary tumor must lie above the peritoneal reflection (i.e., patients with rectal cancer are not eligible); surgeon confirmation that the entire tumor was above the peritoneal reflection is only required in cases where it is important to establish if the tumor is a rectal or colon primary Prior rectal surgery preventing insertion of the TRUS probe. Any rectal pathology, anomaly or previous treatment, which can change acoustic properties of rectal wall or prevent safe probe insertion (e.g., fistula, stenosis, fibrosis). Clinical stage T3 or less (pelvic MRI shows no rectal and ureteral invasion) Clinical stage T4 (pelvic MRI shows rectal and/or ureteral invasion) History of abdominoperineal resection for rectal cancer, rectal stenosis, or other major rectal pathology Phase Ib only:\r\n* Patients with metastatic rectal cancer are allowed if their primary site meets other eligibility criteria and chemoradiotherapy is recommended as initial therapy for symptom palliation by the multidisciplinary treating team\r\n* Patients with locally advanced unresectable rectal cancer are allow provided:\r\n** There is no evidence of recto-vaginal, recto-vesicular, recto-intestinal fisutalization\r\n** Standard dose and schedule chemoradiotherapy is recommended as initial therapy by the multidisciplinary treating team Have pathologically confirmed colon or rectal adenocarcinoma A histologically confirmed rectal cancer with measurable or evaluable disease on imaging or endoscopy Any prior radiation for rectal cancer Rectal/pouch polyposis as a stratification site as follows: For all subjects, any rectal/pouch polyps > 5 mm must be excised at \baseline\. prior significant rectal surgery (hemorrhoidectomy is acceptable; rectal resection/fissure repair are excluded); rectal fibrosis/stenosis; anoscopy or proctoscopy may be performed at the investigator's discretion; anomaly of the rectal anatomy or mucus membrane; anoscopy or proctoscopy may be performed at the investigator's discretion; Be undergoing open, or minimally invasive LAR for the treatment of a rectal or rectosigmoid neoplasm (Subjects with rectal or rectosigmoid neoplasm(s) may be treated with or without neoadjuvant therapy. Long-course neoadjuvant therapy must have been completed ?6 weeks prior to LAR surgery (Day 0). Has previously undergone a rectal resection. Has recurrent rectal or rectosigmoid cancer. Has a diagnosis of locally advanced rectal or rectosigmoid cancer undergoing extended en bloc operations. Has a diagnosis of inflammatory bowel disease (IBD). Subjects with rectal or rectosigmoid cancer neoplasms and IBD are excluded. Histologically confirmed colon or rectal cancer with metastatic disease Contraindication to placement of endorectal MRI coil, biopsy device or ultrasound probe (e.g., severe hemorrhoids, anal fissure, recent rectal surgery, or prior abdominoperineal resection) Evidence of recurrence of rectal cancer prior to the start of study treatment Prostate size on trans-rectal ultrasound (TRUS) measurement less than 50 grams Have had prior major rectal surgery (except hemorrhoids). Receiving chemotherapy (CTX) for breast, colon, rectal, small intestine, or ovarian cancer on a 7, 14, or 21 day schedule with the chemotherapy dose given on day 1 Diagnosis of stage II-III colon or rectal cancer planned for treatment with adjuvant chemotherapy scheduled as part of standard treatment Eligible patients must have been diagnosed with colon or rectal adenocarcinoma Diagnosed with pancreatic, esophageal, rectal, colon, hepatobiliary, or gastric cancer (including patients with prior diagnosis of another cancer) Grade 3 or 4 diarrhea, rectal bleeding, abdominal cramping, or incontinence of stool =< 7 days prior to registration History of rectal cancer or neoplastic polyp (i.e. intramucosal carcinoma, carcinoma in situ) Rectal prolapse Active anal/rectal abscess Intra-operative injuries (for example: rectal injury) Have been diagnosed with stage I-III colon or rectal adenocarcinoma Have not had a clinician-provided digital rectal exam in the prior 3 months Participants must have endoscopically accessible distal colon and/or rectal mucosa (i.e. participants must have at least part of the descending/sigmoid colon and/or rectum intact). Cytologically or histologically confirmed of primary colon or rectal adenocarcinoma with resectable cancer, who have not received any treatments for cancer Known diagnosis of stage III colon or rectal cancer will be excluded from the study Malignancy or mass that is non-gynecologic in origin (mass/tumor of origin other than reproductive organ such as rectal, abdominal, breast) Clinical stage =< T2a by digital rectal exam (DRE) Participants must have endoscopically accessible distal colon and/or rectal mucosa (i.e. participants must have at least part of the descending/sigmoid colon and/or rectum intact) Any history of current or prior rectal cancer Any rectal pathology, anomaly or previous treatment, which could change acoustic properties of rectal wall or prevent safe probe insertion (e.g., stenosis, fibrosis, inflammatory bowel disease, etc). Extracapsular extension suspected on digital rectal exam with confirmation on MRI; suspicion of extracapsular extension on MRI alone is not an exclusion for study enrollment Symptomatic distal rectal stenosis Prior abdominopelvic radiation or radiation for rectal cancer Surgical absence of a rectum or the presence of a rectal fistula Abnormal digital rectal exam (i.e. palpable nodule, induration or firm area to the prostate) A diagnosis of histopathologically confirmed rectal adenocarcinoma Locally advanced disease as determined by endoscopic rectal ultrasound (ERUS) or pelvic MRI; endoscopy reports should clearly state both the T and N stage Prior receipt of any therapy, including local excision, radiation or chemotherapy, for the diagnosed rectal adenocarcinoma Suspected stage ? T2 on rectal examination (organ-confined prostate cancer) within the previous 3 months Diagnosis of rectal cancer Patient with prior rectal surgery or anal stricture requiring surgical intervention prior to biopsy.