This is the expected behavior, because the format dropdown menu is supposed to apply styles to the whole "paragraph" - so, if your EnterMode setting is P, then the format dropdown menu would apply the same style to everything inside a <p>.
If your users are separating paragraphs with two <br>s, then you may want to change the EnterMode setting to 'br' instead.
Re: Bug? Format/style to line without a following <p>
If your users are separating paragraphs with two <br>s, then you may want to change the EnterMode setting to 'br' instead.