EPSG 4326 is totally fine. I have a feeling that whatever was used to collect the GCP’s did not have an accurate base coordinate, and judging by the RMS values an accurate survey either. You should find out if the base coordinate was known and manually entered or derived at the time of the survey. These two methods can be vastly different. If the GCP’s didn’t show up where you expect them when uploaded it should have been an instant clue that was easy to see.