When we factor a polynomial of this form, we are basically reversing this process of multiplying linear expressions together:

\qquad \begin{eqnarray} (x + ay)(x + by)&=&xx &+& xby + ayx &+& ayby \\ \\ &=& x^2 &+& \green{(a+b)}xy &+& \blue{ab}y^2 \\ &\hphantom{=}& \hphantom{x^2} &\hphantom{+}& \hphantom{\green{LINEAR}xy} &\hphantom{+}& \hphantom{\blue{CONSTANT}y^2} \end{eqnarray}

\qquad \begin{eqnarray} \hphantom{(x + ay)(x + by)}&\hphantom{=}&\hphantom{xx} &\hphantom{+}& \hphantom{xby + ayx} &\hphantom{+}&\hphantom{ayby} \\ &\hphantom{=}& \hphantom{x^2} &\hphantom{+}&\hphantom{\green{(a+b)}xy}&\hphantom{+}&\hphantom{\blue{ab}y^2} \\ &=& x^2 &+& \green{LINEAR}xy &+& \blue{CONSTANT}y^2 \end{eqnarray}

The coefficient on the xy term is LINEAR and the coefficient on the y^2 term is CONSTANT, so to reverse the steps above, we need to find two numbers that add up to LINEAR and multiply to CONSTANT.

You can start by trying to guess which factors of CONSTANT add up to LINEAR. In other words, you need to find the values for a and b that meet the following conditions:

\qquad \pink{a} + \pink{b} = \green{LINEAR}

\qquad \pink{a} \times \pink{b} = \blue{CONSTANT}

If you're stuck, try listing out every single factor of CONSTANT and its opposite as a in these equations, and see if it gives a value for b that validates both conditions. For example, since abs(A) is a factor of CONSTANT, try substituting abs(A) for a as well as -abs(A).

The two numbers -A and -B satisfy both conditions:

\qquad \pink{-A} + \pink{-B} = \green{LINEAR}

\qquad \pink{-A} \times \pink{-B} = \blue{CONSTANT}