Support Enzyme through EnzymeRules

[vchuravy]

Took a stab at defining a rule for Enzyme, still need to port the reverse rule

[gdalle]

Hi, thanks for the contribution! I didn't think anyone was actually using this package.
Are you sure you pushed all the commits?

[vchuravy]

sigh missed the extension itself.

I was looking at ImplicitDifferentiation for AD through non linear problems and adjoint models for implicit time integration.

[gdalle]

I was looking at ImplicitDifferentiation for AD through non linear problems and adjoint models for implicit time integration.

Feel free to suggest any improvements!
Right now the biggest limitations are

• the out-of-place linear operator
• the allocating linear solver of KrylovKit (but it was the one that came closest to idiomatic Julia without type restrictions, as opposed to Krylov which has lots of weird conversions)

[gdalle]

I'm not convinced we should define a rule for the version with a prep. The main reason that one exists in the ForwardDiff extension was to enable the use of sparse Jacobian matrices inside the linear system, where the sparsity pattern and coloring are contained in prep. But in theory, including prep in an Enzyme rule means we should define derivatives of the result with respect to prep, which seems complicated?

[gdalle]

Thanks for the contribution! Before I review, can you add Enzyme to the systematic tests, both as an inner and as an outer backend, to see if they run?