Do I need a visa or I can use the VWP (ESTA) to get married to an F1 or H1b visa holder? Is it possible to stay in the USA after that or I have to go back to my European Union country? Thank you.
If you enter on the VWP, you'll have to leave the US after you are married, to apply for an F-2 or H-4 visa. Those who have entered on the VWP are not permitted to extend their stay or change to another nonimmigrant status; see http://travel.state.gov/content/visas/en/visit/visa-waiver-program.html for more information.
(There are exceptions. For example, one can extend one's stay if required to do so by a medical emergency, but that is not relevant to your question. Immediate family members of US citizens can also adjust status despite having entered under the VWP, but that doesn't help you since you are planning to marry a nonimmigrant.)
If you enter on a B-2 or other nonimmigrant visa, you will be eligible to apply to change your nonimmigrant status to F-2 or H-4. Changing status is more expensive, though, so you have to weigh that cost against the cost of travel if you take the VWP route. For more information about changing nonimmigrant status, see http://www.uscis.gov/sites/default/files/USCIS/Resources/C2en.pdf.
The fee for B and F visa applications is currently $160, while the fee for H visa applications is $190 (source: http://travel.state.gov/content/visas/en/fees/fees-visa-services.html). The fee for a change of status application is $290 (source: http://www.uscis.gov/i-539).
Anyway, you should probably discuss this with an immigration lawyer, as there are probably other factors you'll want to consider. A lawyer might be able to advise you on the processing time you could expect with each option, for example.