Before I even start, let me say that I haven't found the perfect hosting company, but eApps comes pretty darn close. They hit all the big points, and only miss a few smaller "nice to haves". This is why I promote them to all my consulting clients and use them exclusively for hosting Reliable Response Notification.
#1 - Reliability
This is more than the server not crashing. I've been on hosting companies that switch around the file system structure without telling me. Since RRN uses a few absolute paths in the config file, moving these paths have negative affects. I've also had hosting companies simply switch off Tomcat, or exceed the allotted maintenance windows, or change IP address. I need to know that my hosting company won't break my app on me.
eApps does require you go through some hoops to get everything working reliably. You need to set Apache directives through the control panel...just editing files is a no-no. I got caught by this. But, they patiently explained how and why it works, so I'm on board.
Also, I don't mind being on a shared server, but there needs to be some protection against the other customers. I've been a bad shared customer at times, too. I had a nasty bug where Yahoo IM access would occasionally drive CPU utilization to 100%. Some hosting companies happily allow you to continue driving CPU up. Others will stop your app and boot you off the server. A good hosting company simply reduces the CPU cycles you have available.
#2 Support -
I *HATE* when someone tells me "I rebooted it and it works now". I need to know why it failed, so I can avoid that in the future. Customer support isn't about answering emails. It's about populating those emails with useful information. Phone support is nice, too. Online chat doesn't work well for me.
#3 Features -
Virtuozzo/OpenVZ is a wonderful thing. I suppose Xen and VMWare are, too. Most of my apps don't require an expensive hosted server. OpenVZ gives me the ability to purchase a small, cheap server that acts like a dedicated OS. Plus, it can dynamically scale, restrict resource usage, and migrate between physical servers.
Linux has a lot of good software for it. Some of that should be available to the hosted server. MySQL, PostgreSQL, Tomcat, Apache, PHP, curl/wget, Nagios, and some web stats package. If you don't support those, then at least allow us to use the standard package installers, yum, apt or whatever, to install them. Failing that, then, we should be able to install from source.
#4 Scalability
With most hosting plans, you can move up and down plans fairly easily, ie scaling vertically. eApps takes it to an extreme. You can move up and down, most of the time without stopping your application. 0 downtime. As well, they support clustering, using excellent hardware load-balancing. This allows you to scale horizontally, by adding new servers. Combine that with their super-cheap, super-flexible OpenVZ servers, and you have one of the most affordable, scalable solutions on the planet.
All in all, I've been very happy with eApps. They have some competition from Amazon, which I imagine will get tougher soon. They also have competition from the classic providers like Verio, GoDaddy and Network Solutions. None of these companies provide the individualized support, attention to your particular app, and general excellence that eApps does.