(1 + a^2)y^2 - 2by + b^2 - ab^2 = 0
Using quadratic formula:
y = [2b +/- sqrt((-2b)^2 - 4(1 + a^2)(b^2 - ab^2))] / [2(1 + a^2)]
= [2b +/- 2sqrt(b^2 - b^2(1 + a^2)(1 - a))] / [2(1 + a^2)]
= [b +/- b sqrt(1 - (1 + a^2)(1 - a))] / [1 + a^2]
= [b +/- b sqrt(1 - 1 + a - a^2 + a^3)] / [1 + a^2]
= [b +/- b sqrt(a^3 - a^2 + a)] / [1 + a^2]
That's more or less about as simple as I can make it. Hope that helps!
EDIT: I confirmed with a CAS that my answer is right from:
(1 + a^2)y^2 - 2by + b^2 - ab^2 = 0
onwards. It seems to be refusing to solve the original system though.
EDIT: Gah! Fixed stupid mistake. Everything works now:
ax + y = b ... (1)
ax^2 + y^2 = b^2 ... (2)
ax = b - y ... (1a)
(ax)^2 + ay^2 = ab^2 ... (2) * a
Substitute (1a) into (2) * a:
(b - y)^2 + ay^2 = ab^2
y^2 - 2by + b^2 + ay^2 = ab^2
(1 + a)y^2 - 2by + b^2 - ab^2 = 0
Using quadratic formula:
y = [2b +/- sqrt((-2b)^2 - 4(1 + a)(b^2 - ab^2))] / [2(1 + a)]
= [2b +/- 2sqrt(b^2 - b^2(1 + a)(1 - a))] / [2(1 + a)]
= [b +/- b sqrt(1 - (1 + a)(1 - a))] / [1 + a]
= [b +/- b sqrt(1 - 1 + a^2)] / [1 + a]
= [b +/- ab] / [1 + a]
= b(1 + a) / (1 + a) or b(1 - a) / (1 + a)
= b or b(1 - a) / (1 + a)
Hope that helps now!