adjust to get CI test cases passed on XPU

[kaixuanliu]

No description provided.

[kaixuanliu]

W/ this PR, we can get following failed cases passed on XPU:

- pytest -rA tests/pipelines/pag/test_pag_kolors.py::KolorsPAGPipelineFastTests::test_float16_inference
- pytest -rA tests/pipelines/kandinsky2_2/test_kandinsky_controlnet.py::KandinskyV22ControlnetPipelineIntegrationTests::test_kandinsky_controlnet
- pytest -rA tests/quantization/gguf/test_gguf.py::SD35LargeGGUFSingleFileTests::test_pipeline_inference
- pytest -rA tests/pipelines/ledits_pp/test_ledits_pp_stable_diffusion.py::LEditsPPPipelineStableDiffusionSlowTests::test_ledits_pp_editing

[kaixuanliu]

@a-r-r-o-w @DN6 , Hi, we are deprecating IPEX in our software stack. Hence there will be some precision variance for model output in test cases. pls help review, thx. @yao-matrix FYI.

[kaixuanliu]

@bot /style

[DN6]

@bot /style

[github-actions]

Style fixes have been applied. View the workflow run here.

[kaixuanliu]

Hi, @DN6 , Can you help merge?

[yao-matrix]

@a-r-r-o-w , could you help merge it? Thx

[a-r-r-o-w]

Started the tests, will merge as soon as CI is green. Thanks!

[a-r-r-o-w]

Hmm, there seems to be some failures for tests that were already fixed in main. Let me try to merge and run again

[a-r-r-o-w]

Failing test seems to be unrelated to changes in this PR

[sayakpaul]

This seems to cause this failure on a CUDA device:
https://github.com/huggingface/diffusers/actions/runs/15890953880/job/44813455615?pr=11705#step:8:943

@kaixuanliu could you please take a look?

[kaixuanliu]

@sayakpaul Sure, will try to fix.