Has had a prior monoclonal antibody within 4 weeks prior to study Day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events (AEs) due to agents administered more than 4 weeks earlier; non-clinically significant adverse events may be considered as an exception after discussion with and approval by the principal investigator SAFETY RUN-IN: Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier RANDOMIZED PHASE II CLINICAL TRIAL: Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior anti-cancer monoclonal antibody within 4 weeks prior to study day 1 or has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to a previously administered agent Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Prior anti-cancer monoclonal antibody =< 28 days prior to registration or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Prior treatment with a monoclonal antibody within 4 weeks prior to the first dose of MK-3475 or has not recovered (i.e. =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has received a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., ? grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had prior monoclonal antibody therapy within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events (AEs) due to agents administered more than 4 weeks earlier Has had prior anticancer therapy within 4 weeks of study day 1 or has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Patients who have received itraconazole or fluconazole within 3 weeks prior to registration or those who have not recovered (i.e., back to baseline or grade 1) from adverse events (AEs) due to agents administered more than 3 weeks earlier Subjects must not have received a prior monoclonal antibody within 4 weeks prior to the first dose of study medication, and must have recovered (=< grade 1) from adverse events related to any anti-cancer agent administered > 4 weeks previous to the first dose of study medication Administration of a monoclonal antibody within 4 weeks prior to initiating study treatment or has not recovered (i.e., =< grade 1 or at baseline) from adverse events (AEs) due to agents administered more than 4 weeks earlier Received prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e. =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks or 5 half-lives, whichever is shorter, prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had prior anti-cancer monoclonal antibody within 4 weeks of Study Day 1 or who has not recovered from adverse events due to agents administered >4 weeks earlier Has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to investigational or standard agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks or 5 half-lives, which ever is shorter, prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Subject has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 2 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier; denosumab is allowed as long as not < 1 week prior to study day 1 and not administered on day of MK-3475 infusion Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., ? grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Prior monoclonal antibody within 4 weeks prior to study day 1 or individuals who have not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Subject has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., ? grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier DOSE ESCALATION COHORT: Prior monoclonal antibody within 4 weeks before study day 1 or not recovered (? grade 1 or at baseline) from adverse events (AEs) due to agents administered more than 4 weeks earlier; exception to this rule would be use of denosumab DOSE EXPANSION COHORT: Prior monoclonal antibody within 4 weeks before study day 1 or not recovered (? grade 1 or at baseline) from AEs due to agents administered more than 4 weeks earlier; exception to this rule would be use of denosumab Has had a prior chemotherapy, targeted small molecule therapy, or monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Non-small cell lung cancer patients only: has had a prior monoclonal antibody within 4 weeks prior to first protocol treatment or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events (AEs) due to agents administered more than 4 weeks earlier Prior monoclonal antibody within 4 weeks prior to study Day 1 (2 weeks for RS participants) or who has not recovered (i.e. ? Grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier (2 weeks for RS participants). Prior anti-cancer monoclonal antibody within 4 weeks prior to first dose of study drug or has not recovered from adverse effects due to agents administered more than 4 weeks earlier Has had a prior anti-cancer monoclonal antibody within 4 weeks prior to study Day 1 or who has not recovered (i.e., ? Grade 1 or at baseline) from AEs due to agents administered more than 4 weeks earlier. Prior anti-cancer monoclonal antibody within 4 weeks prior to first dose of study drug, or not recovered from adverse effects due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study Day 1 or who has not recovered (i.e., ?Grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier. Has had a prior chemotherapy, targeted small molecule therapy, or monoclonal antibody (except bevacizumab) within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier; wash out period for bevacizumab is 14 days Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had monoclonal antibody therapy within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 at baseline; excludes alopecia and grade 2 neuropathy) from adverse events due to agent(s) administered more than 4 weeks earlier Participant has had a prior monoclonal antibody within 4 weeks prior to study Day 1 or who has not recovered (i.e. ? Grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier. Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events (AEs) due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to prior therapies Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Phase Ib: Received prior monoclonal antibody therapy other than bevacizumab within 4 weeks of study registration or who has not recovered (i.e., ? Grade 1 or at baseline) from adverse events of such agents administered more than 4 weeks earlier. Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to radiation therapy or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier; denosumab is a prohibited medication on study and for 4 weeks prior to day 1 Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Patient has had a prior monoclonal antibody for a non–cancer therapy indication within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events (AEs) due to agents administered more than 4 weeks earlier Has had prior monoclonal antibody therapy within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events (AEs) due to agents administered more than 4 weeks earlier Prior monoclonal antibody within 4 weeks before study Day 1, or has not recovered from adverse events due to agents administered more than 4 weeks earlier. Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from AEs due to agents administered more than 4 weeks earlier Treatment with an anti-cancer monoclonal antibody within 4 weeks prior to day 1 or has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e. =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks before study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from AEs due to agents administered more than 4 weeks earlier\r\n* Note: the following will not be exclusionary: patients may have any grade alopecia or lymphopenia and still participate if other inclusion/exclusion criteria are met; patients may have grade 1 or 2 neuropathy at baseline and still participate if other inclusion/exclusion criteria are met Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier Prior treatment with a monoclonal antibody within 4 weeks prior to enrollment or who has not recovered (i.e. ? Grade 1 or baseline grade) from adverse events due to agents administered more than 4 weeks earlier. Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =<, grade 1 or at baseline and other than alopecia) from adverse events due to agents administered more than 4 weeks earlier; exceptions to these criteria may be allowed at the discretion of the UNC PI for toxicities that are not expected to be exacerbated by pembrolizumab or nab-paclitaxel Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier