Inclusion Criteria:
* Patients must have histologically or cytologically confirmed, HER2 negative metastatic breast cancer that is historically GD2 expressing (e.g., triple negative breast cancer, metaplastic breast cancer, high grade estrogen positive breast cancer) with available archival tissue. Well differentiated neuroendocrine tumor (NETs) are not eligible for this trial since GD2 expression is unknown. GD2 expression is not required for eligibility but a primary tumor paraffin block is required at enrollment for assessment of GD2 expression
* Patients must have measurable disease, per Response Evaluation Criteria in Solid Tumors (RECIST 1.1) for the evaluation of measurable disease
* Patients must have received at least one prior treatment for metastatic disease and progressed on treatment or been intolerant to treatment
* Female or male \>=18 years
* Eastern Cooperative Oncology Group (ECOG) performance status 0 or 1
* Absolute neutrophil count \>= 1,500/mcL (\> 1.5 X 10\^6/L)
* Platelets \>= 100,000/mcL
* Hemoglobin \>= 9 mg/dL (transfusion to obtain hemoglobin \>= 9 mg/dL within 24 hours prior to dosing is allowed)
* Serum creatinine =\< 1.5 X upper limit of normal (ULN) or measured or calculated creatinine clearance (CrCl) \>= 60 mL/min for participant with creatinine levels \> 1.5 X institutional upper limit of normal (ULN) (Glomerular filtration rate \[GFR\] can also be used in place of creatinine or CrCl)
* Patients must have adequate hepatic function, defined as aspartate aminotransferase (AST) and alanine aminotransferase (ALT) levels =\< 3 X ULN and total bilirubin \< 1.5 X ULN, unless known diagnosis of Gilbert's syndrome, where bilirubin =\< 5 mg/dL will be permitted. Gilbert's syndrome will be defined as elevated unconjugated bilirubin, with conjugated (direct) bilirubin within the normal range and less than 20% of the total. Total bilirubin will be permitted up to 5 mg/dL, if patients have historical readings consistent with the definition of Gilbert's syndrome prior to entering study. Adequate hepatic function for patients with known liver metastases is defined as AST and ALT levels ≤ 5 X ULN
* The effects of the trial agents on the developing human fetus are unknown. However, gemcitabine is known to have negative fetal effects. For this reason: Women of childbearing potential must agree to use adequate contraception at study entry, for the duration of study participation and for 7 months after the last dose of study medication based upon estimated half-life or receptor occupancy. Adequate contraception is defined as 2 highly effective methods of contraception (including a physical barrier). Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Men should refrain from fathering a child using adequate contraception or donating sperm during the study and for 6 months after the last dose of study medications. Adequate contraception is defined as 2 highly effective methods of contraception (including a physical barrier)
* Patients with well-controlled human immunodeficiency virus (HIV) infection are eligible for trial as long as:
* On an effective anti-retroviral therapy (ART) ˃ 4 weeks and with evidence of viral-suppression as defined as HIV viral load ˂ 400 copies/mL within the last 3 months
* CD4 \> 200 cells/µL within the last 3 months; and
* No reported opportunistic infections within 6 months prior to enrollment, except for the following which will be allowed:
* Esophageal candidiasis treated within last 6 months or currently improving with antifungal treatment.
* Oral and/or genital herpes simplex virus (HSV) treated within last 6 months or currently improving with antiviral treatment.
* Mycobacterium avium infection in last 6 months or that has been treated for at least 1 month
* Patients with evidence of chronic hepatitis B virus (HBV) infection are eligible for trial as long as the HBV viral load is undetectable on suppressive therapy, if indicated.
* Patients with history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable or unquantifiable HCV ribonucleic acid (RNA) 12 weeks or longer after definitive treatment completion.
* Patients must be able to understand and willing to sign a written informed consent document
* Any adverse events subjects have experienced from prior therapy must have resolved to =\< grade 1 according to National Cancer Institute Common Terminology Criteria for Adverse Events (NCI CTCAE) version 5
Exclusion Criteria:
* Patients who within 3 weeks prior to study enrollment who have received chemotherapy, investigational agents, or radiation
* Patients with active brain metastases or leptomeningeal metastases are excluded from this clinical trial because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events. However, patients with treated brain metastases are eligible if there is no magnetic resonance imaging (MRI) evidence of progression for 6 months after treatment is complete and within 28 days prior to the first dose of trial drug. Patients requiring immunosuppressive doses of systemic corticosteroids (\> 10mg/day prednisone equivalent) for palliation are excluded
* Patients with a history of another invasive malignancy \< 3 years prior to enrollment (patients with non-melanoma skin cancers, carcinoma in situ of the breast or cervix are eligible)
* History of allergic reactions attributed to compounds of similar chemical or biologic composition to agents used in study
* Uncontrolled intercurrent illness including, but not limited to, ongoing active infection that requires systemic treatment with ongoing antibiotics (eligible if can stop antibiotics on day of enrollment), unexplained fever (temperature \> 38.1˚celcius \[C\]) within 7 days of initial treatment, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situation that in the opinion of the primary investigator would prohibit the patient from complying with study requirements
* Patients with bone metastases who have initiated denosumab or a bisphosphonate therapy within 28 days prior to or after cycle 1 day 1 due to the potential for flu-like symptoms and mild cytokine release syndrome with the initial dose. However, continuation of prior therapy is allowed
* Patients should have no evidence of being immunocompromised as listed below:
* Active, known or suspected autoimmune disease. Patients are permitted to enroll if they have vitiligo, type I diabetes mellitus, residual hypothyroidism due to an autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment or conditions not expected to recur in the absence of an external trigger in the opinion of the primary investigator
* Altered immune function that in the judgement of the PI that may affect a patient's ability to adequately engage the immune system and respond to the immunotherapy agents being administered, including but not limited to: inflammatory bowel disease; active infectious enteritis; eosinophilic enteritis; lupus erythematous; ankylosing spondylitis; scleroderma; multiple sclerosis. These criteria do not include all disease with an immune-related component but are not autoimmune in nature or have a primary alteration in the general immune function that may interfere with the vaccine mechanism of action, for example celiac disease
* Immunosuppressive therapy post-organ transplant
* Concurrent use of chronic use of systemic steroids, except for physiologic doses of systemic steroids for replacement, defined as 10mg of prednisone per day or equivalent, or local (topical, nasal, ophthalmic or inhaled) steroid use or prior concomitant use with chemotherapy. Systemic steroids must have been discontinued \>2 weeks prior to trial start. Prior use of corticosteroids in short-term schemes (duration shorter than 3 days) for indications such as prophylaxis of reactions to intravenous contrast for imaging studies or chemotherapy-related adverse events (AEs) are not considered part of this exclusion. Prior use of corticosteroids for brain metastasis ending at least 14 days prior to enrollment is not considered part of this exclusion criteria
* Pregnant and breastfeeding women are excluded from this study because of the potential for teratogenic or abortifacient effects with all of the agents involved in this trial. Females of childbearing potential who are pregnant, breast feeding, intend to become pregnant, or are not using adequate contraceptive methods or males who are not using adequate contraceptive methods. Women of childbearing potential must agree to use two methods of adequate contraception at study entry be used for the duration of study participation and for at least 7 months for women and 6 months for men after the final dose of any study-related medications
* Clinically significant cardiomyopathy, coronary disease, myocardial infarction, chronic heart failure (CHF) (New York Heart Association class III or IV or hospitalization for CHF), ejection fraction \< 50% or cerebrovascular accident within 6 months prior to enrollment
* Patients with a history of myocarditis are excluded due to the potential of myocarditis with anti-PD-L1 antibodies or other immunotherapies
* Patients who have received any live vaccines within 30 days prior to enrollment (inactivated vaccines including COVID vaccines are allowed)
* Any other condition, which would, in the opinion of the principal investigator the subject is a poor candidate for the clinical trial or would jeopardize the subject or the integrity of the data obtained
High burden on patient
Caution: ClinicalTrials.gov appears to have newer eligibility criteria than the version saved here. Review the current criteria in ClinicalTrials.gov.