The best way is to directly contact the Ausländerbehörde responsible for your area of residence. (I have personally had good and straightforward responses from them in the past when I emailed them.)
That said, as long as your Blue Card remains valid (and you carry your valid passport just in case), I do not see why you cannot travel.
One thing I would double-check is the piece of paper they gave you with your Blue Card. Printed on it should be any conditions attached to your Blue Card. During the first 2 years, that paper should tell you that you have to notify them of any employment changes. Since your Blue Card status is over 2 years old now, you are probably free from that obligation.
This link contains some FAQs about residency permits, most of the information you probably already know. However, the last question contains some contact information of the Bundesamtes für Migration und Flüchtlinge (BAMF). You might also find direct answers from them, in addition to your local Ausländerbehörde.
As a last note, since you have over 33 months with your Blue Card, perhaps you might consider applying for a settlement permit, which may make things even easier.