This extends the approach used for debian-based packages in 3f805bc64e28
to rpm as well. Notable change for both deb and rpm packaging is to use
CFLAGS as defined in the build/Makefile, and not pass them from the
environment which might not be there (as is the case for rpm).
While at it, stop passing CFLAGS in the install phase, as it should no
longer invoke builds (see d54af163c46b).
The rpm part was overlooked in 7a6405566c0, since testing was not done
on the platforms where problem manifested itself, notably Amazon Linux
2023 and Fedora 38+.