Prison bosses at Rye Hill lost a number of staff after it swapped its mix of prisoners to house only men convicted of sex crimes.
The swap was carried out last year but an independent report just published has revealed some staff at the time were unhappy at the changes.
The Independent Monitoring Board said: “The re-role has led to some loss of staff who felt they would rather not work in a sex offender prison.”
The change over started in April 2014 and was completed on July 3 2014, at which point Rye Hill held 564 Category B sex offenders.
The original operating capacity of 625 was reached by 31 July 2014.
During the decant process 455 prisoners were transferred from and to the establishment.