I would like to start with some basic definitions, so we all have the same understanding.
Semitic. This word has two, related definitions:
Relating to or denoting a family of languages that includes Hebrew, Arabic, and, Aramaic.
Relating to the peoples who speak Semitic languages, especially Hebrew and Arabic.
Based on this generally-accepted definition, we can see that it isn't only Jews who are victimized by anti-Semitism. Hostility or prejudice against Arabs, many of whom are Muslims, is also anti-Semitism.