If you want very high longevity on battery, for a device that is not used for high performances graphics, but as an SSH terminal or local editor (but not for kernel compilation on device...), there are dedicated low power CPU like the SAMA5 from Atmel (now Microchip).

Mainline linux support is quite good (Free Electrons is in charge of it), Qt is available, and consumption is below 150mW at full speed (which is only 500/600MHz...) and can be lower a lot in idle to 300uW (from website: http://www.atmel.com/products/microc...ARM/sama5.aspx)

It can't compete with a raspberry pi 3, and even less with a Galaxy S8, performances wise, but for a "linux hand-held" machine, it would allow 26h of operating at full speed on a single standard alkaline AA battery (1.5V, 2600mAh), and months of idle time. Don't forget to take into account screen/wifi/auxiliary consumption however, as they would be the one consuming the most...

Freescale's imx6/7 range could be another way to do it with a more powerful chip (some have several cores, and GPU, but are still quite low power).