We have,

${\left({\tan }^{-1}x\right)}^2+{\left({\cot }^{-1}x\right)}^2=\frac{5{\pi }^2}{8}$

$\Rightarrow {\left({\tan }^{-1}x\right)}^2+{\left({\cot }^{-1}x\right)}^2+2{\tan }^{-1}x{\cot }^{-1}x-2{\tan }^{-1}x{\cot }^{-1}x=\frac{5{\pi }^2}{8}$

$\Rightarrow {\left({\tan }^{-1}x+{\cot }^{-1}x\right)}^2-2{\tan }^{-1}x{\cot }^{-1}x=\frac{5{\pi }^2}{8}$

$\Rightarrow {\left(\frac{\pi }{2}\right)}^2-2{\tan }^{-1}x{\cot }^{-1}x=\frac{5{\pi }^2}{8}$

$\Rightarrow \frac{{\pi }^2}{4}-2{\tan }^{-1}x{\cot }^{-1}x=\frac{5{\pi }^2}{8}$

$\Rightarrow \frac{{\pi }^2}{4}-\frac{5{\pi }^2}{8}=2{\tan }^{-1}x{\cot }^{-1}x$

$\Rightarrow \frac{2{\pi }^2-5{\pi }^2}{8}=2{\tan }^{-1}x\left(\pi -{\tan }^{-1}x\right)$

$\Rightarrow \frac{-3{\pi }^2}{8}=2\pi {\tan }^{-1}x-2{\left({\tan }^{-1}x\right)}^2$

$\Rightarrow 2{\left({\tan }^{-1}x\right)}^2-2\pi {\tan }^{-1}x-\frac{3{\pi }^2}{8}=0$

${\tan }^{-1}x=\frac{2\pi \pm \sqrt{4{\pi }^2+4\times 2\times \frac{3{\pi }^2}{8}}}{2\times 2}$

${\tan }^{-1}x=\frac{2\pi \pm \sqrt{4{\pi }^2+3{\pi }^2}}{2\times 2}$

${\tan }^{-1}x=\frac{2\pi \pm \sqrt{7{\pi }^2}}{4}$

${\tan }^{-1}x=\frac{2\pi \pm \sqrt{7}\pi }{4}$

$Now,$

$x=\tan \frac{2\pi \pm \sqrt{7}\pi }{4}$

It is quadratic equation