Unfortunately I can not login with GDM using X. I am having the same bug as in here :/ For wayland, I have both read/write permissions for my username.
Also I can use the PS5 controller as a mouse just fine. Another symptom I see is when I run xdotool mousemove 150 150, it will print Warning: XTEST extension unavailable on '(null)'. Some functionality may be disabled; See 'man xdotool' for more info.. The null here is concerning to me. Not sure how it says on other system.

Thank you for all your help! I've finally found out what I did wrong. Gnome has it own way to enable/disable an extension which I followed here because of another bug. It is so annoying I am thinking about switching to lightdm permantly.
Thank you very much once again!