Find the indefinite integral x^3 sqrt(x^2 -25)dx using the substitution x=5sec

x = 5secθ
dx = 5tanθ secθ dθ

∫ x ³ √(x ² - 25)dx = ∫ 125sec³θ √(25sec²θ - 25) 5tanθ secθ dθ
= ∫ 625sec⁴θ √[25(sec²θ - 1)] tanθ dθ
= ∫ 625sec⁴θ √[25(tan²θ)] tanθ dθ
= ∫ 3125sec⁴θ tan²θ dθ
= ∫ 3125sec²θ (tan²θ + 1) tan²θ dθ

Now sub u = tanθ
du = sec²θ dθ

= ∫ 3125 (u² + 1) u² du
= ∫ 3125 (u⁴+ u²) du
= 3125(u^5/5 + u³/3) + C
= 625u³(3u² + 5)/3 + C

Back sub
u = tanθ

= 625tan³θ (3tan²θ + 5)/3 + C

x = 5secθ
secθ = x/5 ----- Adj 5, Hyp x, Opp √(x² - 25)
tanθ = √(x² - 25)/5


= 625(√(x² - 25)/5)³ (3(√(x² - 25)/5)² + 5)/3 + C
= 5 √(x² - 25)³ (3(x² - 25)/25 + 5)/3 + C
= 5 √(x² - 25)³ (3x² + 50)/75 + C
= √(x² - 25)³ (3x² + 50)/15 + C

Do you have to use that substitution?

let u = x^2 - 25, then x^2 = u - 25

and du = 2x dx ---> du / 2 = x dx

write

x^3 sqrt(x^2 -25)dx

= x^2 sqrt(x^2 - 25) (x dx)

Then insert our above expressions,

= (u - 25) sqrt(u) (du / 2)

= (1/2) (u - 25) sqrt(u) du

= (1/2) ( u^{3/2} - 25 u^{1/2} ) du

integrate

= (1/2){ (2/5) u^{5/2} - 25 (2/3) u^{3/2} } + C

= (1/5) u^{5/2} + (25/3) u^{3/2} + C

= (1/4) (x^2 - 25)^{5/2} + (25/3) (x^2 - 25)^{3/2} + C............[Ans.]