I wonder whether prompt caches would be the perfect use case of something like Optane.

It's kept for long enough that it's expensive to store in RAM, but short enough that the writes are frequent and will wear down SSD storage