I want to name a new diatom species in the honor of Prof. Vodyanitskiy. What will be the species epithet: "vodyanitskiya", or other? Are there any rules that regulates how the end of such epithets is changed?
The correct form of the specific epithet depends on the generic name (masculine, feminine, and neuter generic names will each require the corresponding, grammatically correct, form of the specific epithet), and on the type of the specific epithet itself:
Thalassiosira bradburyi is named after J.P. Bradbury. Thalassiosira hendeyi is named after N. I. Hendey. The letter "i" is added to the person's last name in both cases. Hence, Thalassiosira species named after V. A. Vodyanitskiy should be, in my opinion, Thalassiosira vodyanitskiyi.
You should check the article 60 of ICN: http://www.iapt-taxon.org/nomen/main.php?page=art60
See in particular Recommendation 60C about specific epithet, and in the specific case, the rule 60C.1 (specific epithet derived from a person name).
You can use the genitive case (as Plant of Vodyanitskiy) as in rule (a), which is independent of generic name, so if the professor is male: X. vodyanitskiyi, and she is female: X. vodyanitskiyae.
or as adjective (so inflected according the genera), in rule (c): X. vodyanitskiyana, X. vodyanitskiyanus, X. vodyanitskiyanum
Note: there are other rules to have a valid published name, and it is recommended to explain why you choose such name. In additional to the ICN code, there is the companion book: The code decoded. A user's guide to the International Code of Nomenclature for algae, fungi, and plants(Regnum Vegetabile 155), by Nicolas Turland