For the USPS:
Priority Mail International from the US to the UK would be between $201.20 and $231.55 for a 66 pound (30 kg) package, the maximum weight that can be shipped, with two of them required to ship 60 kg. The price you pay depends on how you purchase the postage. It's highest at the post office (the Retail rate), lower online (the Commercial Base rate), and lowest if you have a contract giving you Commercial Plus pricing.
First Class Package International service is generally lower priced, but is limited to 64 ounces (1.8 kg). Many individual books weigh more than this, making this generally impractical.
Great Britain restricts M-bags to 20 kg each. The current price for an ISAL M-bag is $2.12 per pound, making the total $279.84, and the 60 kg would have to be split across three bags.
UPS and FedEx probably aren't worth looking into. When I did time and cost for a 60 kg package from the US to the UK, I got quotes of more than $800 from each of them for their lowest-priced services.
It appears that the ISAL M-bag is indeed the cheapest option. Though this may take several weeks to arrive. If you want them to arrive sooner, Priority Mail international, at a bit over $400 and about a week to arrive, may also be an alternative.
Since you also said you wanted insurance, you should be aware that insurance is not available for the M-bag. It is available for Priority Mail International, at a rate of $10.50 for up to $700 cover, plus $1.15 for an additional $100 cover, up to a maximum $5000 (per package) cover.
If I were you I would ship the less important books via ISAL M-bag, and the more important ones that you want to insure via Priority Mail International with appropriate insurance.
All prices were quoted from USPS Notice 123 effective April 7, 2014, and may have changed by the time you read this.