I don't think it's size as an issue, but the impression of it being slow could be due to the navigation of complex folder structures over smb. I have 46 TB of media served by Jellyfin but only 5,000 files or so altogether, but it's also local to the server.
I don't know if there's a way you could try running it local, rather than over SMB?