Yes, but you need to go through the appropriate application process. It's not an automatic right that doesn't require any paperwork.
Spouses of US citizens are eligible for Green Cards (permanent residence). There are certain things that will make you ineligible, such as having a communicable disease or having been convicted of certain crimes, but generally spouses get approved for Green Cards unless there are unusual circumstances. However you do have to go through the appropriate procedure to get the permission to get the Green card. Also the process is very open to immigration fraud (a situation which is a staple of American sitcoms), and because of this immigration tends to scrutinize marriages quite closely.
You are also permitted to marry while in the US. You are then permitted to apply for "adjustment of status", which means you can apply for the Green Card and stay in the US while it is being processed.
However while you are allowed to marry while you are in the US, you are not allowed to travel to the US for the purpose of getting married and remaining in the US. If immigration suspects you are doing that, they are entitled to refuse you admission. This is part of the utter weirdness that is US immigration law.
Alternatively, if you are certain that you want to marry, then you can apply for a K-1 fiance visa, sponsored by your partner. This is easier to get than a general immigration visa. It allows you to marry once you reach the US and to apply for adjustment of status to remain there.
Because this is so complex I strongly recommend you hire an immigration lawyer to sort this out. They will guide you as to which route is better for you.
EDIT: To answer the question " if I was staying in the US for a long time, and then decided to get married would that be considered evidence that I had travelled to get married" - I don't know, and that's why it's a good idea to get a lawyer. US immigration law is very complicated.