Interestingly, this has occoured to me too.
There are some Cortex-A5 chips out there (eg. NXP Vybrid, but SAMA5 is such a chip too) which look hacker-friendly enough to actually design a custom board (or use a compute module developed by somebody) and maybe it's even possible to create some sort of a device with it.

Would be a cool electronics project. (The Vybrid even has a QFP package IIRC, so it could even be hand-soldered.)

However, sadly I came to the conclusion that the resulting device would not be too interesting because of the very low performance of the Cortex-A5 comparted to anything you can find in a smartphone or tablet these days.

Exactly.

UX: neither the Vybrid nor the SAMA5 has serious graphics acceleration. Which means you won't have a smooth GUI either, so you won't have groundbreaking UX either. (The Vybrid has OpenVG which nobody cares about, but even if they did, it wouldn't be on par with a proper GPU.)

Maybe such a piece of hardware would be suitable for something like an e-book reader and some light terminal usage, but I'm not even sure if it could render websites with acceptable speed.

Maybe yes, maybe no. Using an low-performance chip does not alone warrant that the device will have a low power consumption. Neither the Vybrid nor the SAMA5 is meant for this use case. Oh, and implementing power management is a bitch. And the SoC is not the only thing on board which consumes power...

Yes, using a "proper" SoC (imx, omap, whatever) would more likely result in an interesting device. However looking at how the Neo900 guys struggle with it, I'd say this is not viable for a one-man or small-team electronics project.